Virtual Production Explained — Is This The End of the Green Screen?

Over the past three decades, films have increasingly relied on computer-generated imagery to create spectacular visuals that were once unimaginable. However, this reliance has often resulted in sets that bear little resemblance to the final images we see on the big screen. Thankfully, a new technique called virtual production is revolutionizing the way filmmakers work, allowing them to witness in real-time what their shots will look like in their final, polished form. But how exactly does this groundbreaking tool work, and what does it mean for the future of filmmaking?

A Brief History of Virtual Production

Virtual production is a process where real-world filmmaking and digital effects occur simultaneously, unlike traditional green screen filmmaking where the cast and crew had to visualize the final shot in their minds. Its roots can be traced back to techniques like rear projection, where an image was projected onto a screen from behind to create driving or tracking scenes without complicated location work. Stanley Kubrick’s “2001: A Space Odyssey” also used front projection, beaming backgrounds onto a screen from the front for sharper, more saturated images.

At the same time, keying techniques like the green screen became increasingly popular, allowing filmmakers to remove the colored backdrop and insert digital environments more seamlessly. However, for productions heavily relying on green screens, it remained challenging to visualize the final shot, making tasks like framing and lighting more difficult without real reference points.

To solve this issue, James Cameron and his team developed the Simulcam for “Avatar,” a device that combined live footage with computer graphics in real-time, enabling the cast and crew to better understand the digital environments they were working with. This was one of the earliest iterations of virtual production.

How Virtual Production Works Today

The most popular form of virtual production today involves building sets out of massive LED walls that can be programmed to display photorealistic 3D worlds, often created using Unreal Engine, a real-time 3D creation tool originally developed for video games. Unreal Engine’s Nanite technology allows for photographic sources to be imported into its 3D environment, resulting in near-photorealistic backgrounds that require significantly less effort to create than traditional visual effects.

Before filming, the on-set camera is synced with a virtual camera within Unreal Engine. As the camera moves during filming, its movements are tracked, and the background on the LED walls adjusts accordingly, creating a parallax effect that makes the live-action elements and virtual backgrounds appear to exist in the same space. This is a significant improvement over traditional rear and front projection methods, which couldn’t adjust to camera movements, requiring the camera to remain static to avoid distortion.

The LED walls used in virtual production can be massive. On “The Mandalorian,” one of the first projects to utilize this technique extensively, the LED set, referred to as “the volume,” stands 20 feet tall, 75 feet in diameter, and covers 270 degrees of the set. While large-scale LED volumes like this are typically reserved for big-budget productions, companies like Arwall are making virtual production more accessible to indie filmmakers with sub-$10,000 packages that can turn a small screen or studio into a virtual set.

Built-In Lighting

One of the most groundbreaking aspects of LED virtual production is its use as a light source. With traditional green screens, cinematographers had to approximate the lighting of a location, often leading to inconsistencies or unrealistic results. With LED walls, however, the lighting can be far more accurate since the screen itself emits real light from its virtual location.

As “Mandalorian” production designer Doug Chiang explains, “We get natural lighting from the screens that actually are reflected on the characters themselves and that created real reflections in the water. And when you add all that up as much as you can get in camera, the reality of it just enhances and makes it more believable.”

This real lighting also means that atmospherics like rain or smoke can be used for real in a scene and will interact with the light realistically, something that was often challenging or impossible with green screens.

However, while LED screen lighting can be incredibly accurate, it’s not a perfect substitute for being in a real environment. Cinematographers must account for light falloff, which refers to how light dissipates depending on its distance from a subject. The LED screen is not an infinite distance away, so the light falloff will be different than if the light source were millions of miles away, like the sun.

Additionally, LED screens struggle to imitate the look of direct sunlight, excelling instead at diffused, soft light. This is why much of the first two seasons of “The Mandalorian” took place in overcast environments, as director John Favreau noted that “hard daylight is best done in hard daylight.”

While not a perfect one-to-one substitute for real-world lighting, LED virtual production does offer cinematographers far greater control over light than they would have in actual exteriors, particularly for challenging scenes like sunsets or sunrises.

Flexible Locations

One of the most significant advantages of virtual production is the flexibility it offers in terms of locations. As ILM visual effects producer Marissa Gomez explains, “You could switch between the Iceland location to the desert location all within the same day of shooting.” The rotating stage also allows for quicker setups, as you can “just change the background or turn around on an actor. You’re not physically moving the cameras, you’re actually just moving the background. And all the lights change.”

However, there is one notable drawback: limited camera movement. As cinematographer Roger Deakins notes, “If you’re shooting up film totally on stage with sets, we still can’t have a guy run 100 yards. It’s true. Can’t track behind them – there’s no way of doing that. So in a way, there’s a few more technical restrictions.”

While not a perfect substitute for a real location, virtual production can work wonders for projects with globetrotting ambitions, and it also reorders the traditional visual effects workflow. With green screens, 3D renderings of locations were typically created in post-production. With virtual production, these virtual effects must be completed in pre-production to be ready for filming on the LED walls.

As cinematographer Matt Workman explains, “We’re gonna see previs and typically post-production asset creation – that’s all coming up front and it starts to happen really quickly in the process. And I think that the spectrum and the flow chart of when do you do what is very much evolving and changing.”

If the background isn’t fully complete before filming, the LED can still be used by having a green screen on the LED behind the subject.

Acting in a Virtual World

Having the virtual locations visible on set also has significant ramifications for actors. Green screen acting can be challenging, as actors must imagine the space they’ll be in once the scene is complete. With LED sets, actors are almost completely surrounded by visuals of the location, making it easier for them to react naturally to their environment.

As actor Carl Weathers notes, “You see the scenery, you see the environment. It’s not a green screen. It’s not a tennis ball. You don’t have to work hard to envision it. It’s there and you just get to feel it and experience it and enjoy it and then create magic with it.”

Virtual production also solves the practical problem of eye lines. When imagining where objects will be in post, an actor may find themselves looking in the wrong spot, giving away that they’re not in the same space as whatever has been added. With the LED set, what an actor is supposed to look at is actually there, allowing for more natural and believable performances.

Conclusion

With all of these advantages, it’s hardly surprising that virtual production is taking Hollywood by storm. The system allows for quicker setups in a controlled environment while often looking more realistic than a green screen. While virtual production requires more pre-production planning than ever before, it’s already proving to be a powerful tool in the hands of skilled filmmakers.

As with any new technology, virtual production is only as effective as the artists who wield it. But for a technique that’s barely more than a decade old, it’s already revolutionizing the way movies are made, offering a level of control and flexibility that was once unimaginable. While it may not completely replace traditional filmmaking methods, virtual production is undoubtedly shaping the future of cinema, and its potential is only beginning to be explored.