An interview from Epic Games’ Virtual Production Field Guide.
A couple of years ago, Epic Games published the Virtual Production Field Guide. It was a mix of stories and interviews with major innovators in the virtual production space, and of course discussed how Unreal Engine had been used in several projects.
Now Epic is back with Volume 2 of the Virtual Production Field Guide. It includes pieces with directors Jon Favreau and Rick Famuyiwa, Netflix’s Girish Balakrishnan and Christina Lee Storm, VFX supervisor Rob Legato, cinematographer Greig Fraser, Digital Domain’s Darren Hendler, DNEG’s George Murphy, Sony Pictures Imageworks’ Jerome Chen, ILM’s Andrew Jones, Richard Bluff, and Charmaine Chan.
Epic Games has given befores & afters an excerpt from the Field Guide, which is an interview with virtual production supervisor Clint Spillers. He’s worked at Weta Digital, Digital Domain, DreamWorks, and ILM on films such as Avatar, The Adventures of Tintin, Maleficent, The BFG, and Ready Player One. Spillers was also the virtual production supervisor and oversaw the virtual art department for Seasons 1 and 2 of The Mandalorian. Here’s the interview with him.
Can you talk about your role on The Mandalorian?
Clint Spillers: Our goal was to set up a virtual art department, which would serve as a digital construction department. It’s under the guidelines of the production designer regarding the design and the aesthetic. You have people that were building sets on stages and determining how the set would be decorated.
Then we would do virtual scouts with all of the departments coming together, including the director of photography, the visual effects supervisor, set decorators, construction department, art department, and key creatives. We’d make very informed decisions on blocking, lighting, staging, set dressing, etc. based on a host of variables and utilizing all the latest technology that Epic has been developing with our virtual production partners.
We’re not just making decisions for things that will be in the LED environment; we’re also making decisions on practical sets. The methodology decisions are a collaboration between the production designer, the DP, and the visual effects supervisor, who each weigh in on the approach to the volume work.
What kinds of artists fill out the virtual art department?
Spillers: A lot of them come from visual effects-heavy backgrounds. They’re model makers, they understand Substance and understand many 3D applications to build assets. We also get many people from the games industry who know how to build and design for games. They know how to optimize those heavy assets and get them to run in real time, at a high frame rate within the engine, so we can scout them.
We also have a dedicated assistant art director who works within that unit. The crossover between the digital build and the physical build has to marry one-to-one because you’re doing something physical and extending it virtually within an LED environment. The department also consists of people who are good at photogrammetry.
We rely a lot on the set decorators to lay out the physical space that needs to continue virtually. So, we need to capture it with photogrammetry and make it a virtual asset and get it into our scene and turn that over to visual effects. We have all these amazing artists at ILM who create the content for the LED screens using the proxy VAD models as building blocks and a guide for the final content. They also finesse the environments with a Star Wars aesthetic.
What does pre-production look like for the virtual art department?
Spillers: Typically, the production designer and I come on anywhere from 30 weeks before shooting. We’re trying to identify key talent to make up our VAD. And then I’ll build out the schedule for that virtual art department and previs department, which we also call our virtual blocking department. We try to have the front end fully staffed from about 26 to 28 weeks out. We aim to start getting stuff over to ILM about 15 weeks out.
Do you take into consideration the camera optics when designing specific environments?
Spillers: We utilized Perforce servers to get everyone connected in the pipeline and share assets between the departments. The previs team knows the layout of the volume or the physical set. So, simultaneously as we’re scouting within the VAD, we’re also blocking within the virtual blocking department.
Then we have a one-on-one with the DPs, so if they’re lensing something in the VAD, we’re making sure that it works with the blocking. The whole thing is more or less techvised in this process. We’re literally on the same floor in one big bullpen, so there’s a line of communication that is constantly going.
How do you manage the seam between the screen and the physical set?
Spillers: Everything that we’re building is dialed in to the inch, or the millimeter, in terms of the physical piece and how it ties to the virtual extension. What are the decks and raw trims that need to be brought in? How big is that stairway? If there’s a catwalk, how does it connect to the virtual catwalk? The earlier we bring people on, the more decisions we can make up front in partnership with the visual effects supervisor, ILM environments supervisor, the DP, and the more successful we are in providing all that guidance to ILM, who is responsible for creating those final assets and making sure that that marriage is one-to-one.
How do you optimize environments for performance on the LED wall?
Spillers: At the front end of the show, we’re trying to make sure that everything is optimized. So, when you’re scouting, it is running in real time and at a good frame rate. Once packaged up and given over to ILM, their team optimizes those assets to work within their engine, which is how the final pixel environments are rendered in real time on set.
Do the tools lend themselves to the current era of remote collaboration?
Spillers: Absolutely, and building teams with Unreal Engine gives us a huge leg up. We lean heavily on Perforce servers for this type of work, but we can also do remote scouts. If I have workstations and headsets deployed to the key creatives, we are 100 percent able to do live sessions successfully, and everyone is remote.
What advice would you give someone looking to start a career in the virtual art department?
Spillers: I tell everyone to get involved with Unreal Engine and look at the tutorials. Get familiar with Maya and Substance Painter and start getting those assets into Unreal. I refer interested people to Epic Games’ website and the Virtual Production Field Guide. Also, study past virtual production projects like Ready Player One, Tintin, Avatar, and The BFG to understand how these workflows have evolved. On top of the tutorials, how-tos, and reading everything you can consume, get into Unreal, and muck around with it under the hood.
Where do you see virtual production evolving next?
Spillers: The Mandalorian is a very technically ambitious show, but you could use virtual production to do something as intimate and low-budget as an indie or a short film. Anyone can get their hands on Unreal Engine. So, instead of sending 20 people out to a location to scout, we can get them scouting in engine remotely and make the decisions in real time.
When we were doing Ready Player One, we would prep scenes that we were building physically on a set. We needed to understand how many pods we needed to build and how many we could extend. We could virtually scout a location and make critical decisions. If we’re never going to see a part of a set, we don’t need to build it and can move that money somewhere else.
More traditional departments are going to step into the virtual art department world. Productions will continue to lean heavily on it. Other filmmakers are contacting me, and they’re not necessarily trying to do an LED wall. They want to build a VAD to help make these decisions upfront, and they want to use the engine to do that. I just see more people getting involved, which is going to start creating more ideas.
Download Volume 2 here.
