The Rise Of Virtual Production — Hollywood’s Most Confusing Miracle

In 1930, Fox entered a new frontier of moviemaking by debuting a brand-new way to show motion on film: rear projection. Used primarily for driving scenes — or that famous airplane scene in North by Northwest — this technique would allow for the actors to be stationary while a projected background behind them showed movement. The effect was rudimentary, but believable, and became a go-to approach for filmmaking for decades after.

Almost a century later, rear projection has evolved into something a bit more complicated: LED walls, often referred to with the umbrella term of “virtual production,” or sometimes, “AR Walls.” Much like rear projection, virtual production allows for actors to remain stationary while an ultra-realistic image is shown on vast LED soundstages. It’s essentially like a giant TV screen, allowing cinematographers, directors, and actors to see the imagery (often referred to as “plates”) that will be shown in the final product.

The mainstream debut of this technology was in 2019, on Season 1 of The Mandalorian, which used Industrial Light & Magic’s StageCraft technology, colloquially called “The Volume.” With the Volume, Pedro Pascal’s Din Djarin could be on Tatooine one minute and Coruscant the next with the touch of a button, potentially saving entire stages of post-production VFX work.

Since that premiere in 2019, virtual production has followed a strange path, soon used proudly on sets everywhere and, recently, seemingly disappearing, if not from the sets themselves, from conversations. The journey of the AR Wall follows a common path of tech in Hollywood. Like rear projection, the hot new thing gets used and overused, sometimes abused, and finally perfected to the point that it becomes an essential but unsung part of the production process. So just what happened? Has virtual production changed Hollywood like it seemed poised to? The answer, much like most of the industry, is all about logistics, and the push and pull of what works versus what actually looks good in the end.

The Muppets Show The Way

After The Mandalorian (and, soon after, Matt Reeves’ The Batman) made this technology the hot new thing, it was actually The Muppets who proved its feasibility for non-blockbuster projects. In 2021, while Hollywood was still majorly shut down, Disney+ made The Muppets Haunted Mansion, a variety special that used the technology heavily. Using the Volume wasn’t in the budget, so this special turned to ARWall, another virtual production studio. “It was a very high-tension shoot,” ARWall CEO Rene Amador tells Inverse. “LA people were dropping by there to be like, ‘Hey, how's it going? Can we tell other people that we could shoot?’”

Despite all the pressure, Amador chalks this special up as a success. “It's, I think, one of my favorite Muppets films ever made. Specifically, the moment of Gonzo just having a sense of his own mortality and stuff, it was like, ‘Damn, I'm tearing up on set.’”

For cinematographers, this new toy came with a plethora of unknowns. When Nikolaus Summerer first used virtual production for the German Netflix series 1899, his research was no different than the average spectator’s. “Basically, the only real research that there was was the making of a video of The Mandalorian and the few articles that described the whole procedure around it,” he tells Inverse. Thankfully, he was also able to speak to Mandalorian cinematographers Baz Idoine and Greg Fraser to get all the technical details. “There were certain things they couldn’t say openly,” Summerer explains on learning about the process of using this tech. “Either that or they didn’t put it in the video.”

Thankfully, this naivete actually ended up being a boon to the production. “We all were really, really curious and open, and that was the best thing about it,” Summerer said. “Now, looking back at it, it was pretty brave to do all this without knowing anything. I think we succeeded because we didn't know; we just asked questions and then approached it in a certain way.”

The Pros of Virtual Production

Virtual production often gets compared to the green- and blue-screen soundstages we see used in blockbusters nowadays, but rear projection is a much more apt comparison, and much like rear projection, it all stemmed from driving scenes. Cinematographer Tobias Datum first used virtual production for a driving scene and a mountaintop scene in the Hulu series Tiny Beautiful Things. Then, he was given the opportunity to shoot an entire episode using virtual production backgrounds on the Doctor Who episode “BOOM,” which was almost entirely set inside a crater on an alien planet, and he used the tech throughout Season 1 of Apple TV+’s Murderbot.

For him, sci-fi is tailor-made for virtual production. Unlike shooting on location, a virtual production can look as otherworldly as possible. “You can create worlds that are not bound to what we encounter here. So that makes it certainly more fun to play with a completely virtual space.” In fact, Amador notes that often, creatives don’t even realize how handy virtual production can really be. “There are so many things that happen on set with this technology where you go, ‘Could we do this? Could this do this little trick shot? Could we move the virtual camera to make it seem like it's doing this thing or could we shrink the scale of it to make the guy look gigantic?’ There are so many little things that you can do with virtual production that you would not write in a marketing brochure.”

Industrial Light & Magic has always made it clear that the Volume stage isn’t a one-stop shop for production. The Mandalorian was often shot on a Volume stage, but in collaboration with other techniques. For example, in one Season 2 scene, the background shown on the Volume screen wasn’t a CGI rendering, but actually footage of a “bigature” production that had been created previously. Then, to add more dynamic elements, stop-motion machinery was added to the background — decades-old VFX technology combined with the real thing.

“Usually we talk with filmmakers about what is the story they're trying to tell, how are they trying to tell it? And in looking at that, you'll start to identify the different tiers of, ‘This is an obvious win.’ It is an incredible source of lighting and reflections as well as refraction. So if you know there's a lot of glass, a lot of liquid, you have vehicles with lots of different reflective surfaces,” Justin Talley, virtual production supervisor at ILM StageCraft, tells Inverse.

Lighting is an incredibly important factor in virtual production. With a typical blue- or green-screen, any kind of reflection or refraction of the keyed-out color has to be meticulously rotoscoped by hand. But with the Volume, the light you see is the light you’ll get, saving work in post and allowing those on set to get a sense of the final product long before post.

That may seem like a minor perk, but it can be a game-changer for the process. “When you put all of your creative leads in the same room at the same time, looking at the same thing, discussing the same thing, and making decisions about that same thing at the same time in real time, that is a very powerful tool,” Talley says.

Of course, that has its drawbacks too — because of the on-the-ground decision-making, there’s much more prep involved for a virtual production. “People always said fix it in post, or we could fix it in post. And then the thing about virtual production is that you fix it in pre,” Jim Rider, virtual production supervisor at Pier59 Studios, a virtual production studio based in New York City, tells Inverse.

But the more this technology was used, the more benefits became clear. As Rhys Ifans pointed out in a House of the Dragon behind-the-scenes clip, virtual production means they can keep a sunset indefinitely. Day scenes can be shot at night, night scenes can be shot in the day. “What I've found when I work with directors is that the thing that they lean into is they're like, ‘I'm just going to do so many more takes,’” Amador says. “Maybe if I were on location, I would've done three takes. So now I'm going to be doing 15 to 20 every single time.”

Even the actors benefit. Without temporary measures for VFX, actors can all have the same eyeline and interact with an object and be fully immersed in the scenario. According to Summerer, this effect was so powerful while shooting scenes with rolling waves that some of the crew even started to feel seasick.

A Gold Rush of Streamlined VFX

In the early 2020s, the Volume became a technology you could brag about, something that appeared over and over in behind-the-scenes featurettes. Summerer referred to it as a “gold rush.”

But slowly but surely, attitudes seemed to shift towards the seamless. Use of virtual production stopped being about showing off what’s possible and started being about sneaking in virtual production elements into traditional projects. Percy Jackson and Olympians used it, as did The Fabelmans. ILM provided the Volume for Steven Spielberg’s semi-autobiographical movie. “They had a transformer explode and sparks go shooting everywhere. And so that's in the plates, but then we can also run an angle grinder to shoot out sparks in front of the camera,” Talley says.

Often, talent safety is a concern just as much as verisimilitude. “In Train Dreams, there's a scene of a forest fire, and the forest fire is all shot in a Volume,” Lorenzo Ferrante, head of virtual production at Pier 59 Studios, tells Inverse. “And watching it, I was like, it just looked great, and I didn't think about it. That's the best use case when you don't even think about it. And then in retrospect, you're like, oh, makes a lot of sense.”

After a few years of the virtual production boom, critcs started to notice a specific “look” to things filmed on a Volume stage: hazy lighting and flat backgrounds. It was hard to put into words, but we all knew it when we saw it. Eventually, a Volume background could become as glaring as a painted background in an Old Hollywood movie. So what causes this “Volume look”? Usually, it’s a disconnect between the background and foreground. For example, “BOOM,” the Doctor Who episode, took place in a crater, so the horizon was never shown, and the orange haziness of the background was just atmospheric. It’s not a mistake, but it is something folks have started to notice.

However, in other projects, there were certain tells that can be interpreted as blatant giveaways, like constantly overcast lighting, since LED walls can’t replicate harsh, direct sunlight. Sometimes, it was a constantly blurry background, which made the foreground seem out of place. But that doesn’t mean it was an example of overuse of virtual production. In fact, it doesn’t mean that that was virtual production at all.

“There are a few very funny pieces online where there's a whole piece about the Volume environments, and not one of the environments they discussed during the piece was a Volume environment,” Talley says. “It was all backlot work where it's like, well, it was an overcast sequence they shot on the backlot!”

If there are some poorly-done usages of the Volume, just the fact that they are noticeable at all proves why they’re not the best. “Just like with any visual effect, I would argue that if it's a well-done visual effect, you wouldn't know that it's a visual effect at all,” Rider says. “That also goes with virtual production. You have absolutely seen many things that have been done in virtual production that you wouldn't know because they're holding together and they're holding together so well.”

So what’s the reason for this “gold rush”? It’s likely that there wasn’t any sort of over-reliance on this tech; it’s just another classic example of the comings and goings of film techniques. “This is a filmmaking technique that everyone saw the development of in real time,” Talley said. How it could be implemented best was still yet to be determined, so it appeared often. “After The Matrix came out, there was a draw toward a certain way to approach color timing or palettes for films.” Hollywood is not immune to fads, but they serve an important role — to find the best circumstances for a new innovation.

A Tool, Not a Magic Wand

So what happened to all of the virtual production shots in your favorite shows? To put it plainly, they got better at hiding. “It's not just about how good the virtual production team is at controlling the wall,” ILM Production Manager Claire Brooks tells Inverse. “It's your DPs and your gaffers and your production designers and your set decorators and props and costumes and all of that coming together. I think the industry is getting better at working with this tool. So it's becoming more integrated, it's becoming less obvious.”

And with more time comes more innovation. “We've simultaneously seen an explosion in just LED technology, forget about virtual production, for lighting, for advertising, you're starting to see some of these same techniques on billboards in Tokyo and in live music performance,” Talley says. “And those are spaces where some of these things continue to find hooks.”

The consensus seems to be that virtual production won’t replace blue- or green-screen effects, but it was never meant to. There are things you can do with virtual production that you can’t do on a green screen, but conversely, there are things you can do on a green screen that are just impossible with virtual production.

Everyone who spoke to Inverse about this technology said the same thing over and over: this is a tool. Like any other tool, it can be used well or it can be used poorly, but with more time, more practice, and more seamlessness, it could become a go-to method in the years to come.