The insanely intricate work behind ‘Rise of Skywalker’s’ stereo conversion

Here’s how Stereo D pulled it off

It still stuns me just how much effort is involved in the stereo conversion. It involves an analysis of the anticipated 3D experience for the audience, and then a whole lot of roto, matte creation and painting to fill in occlusion areas.

One of the latest big releases to receive the stereo conversion treatment was J.J. Abrams’ The Rise of Skywalker. The conversion was completed by Stereo D, led by stereographer Brian Taber.

I got the chance to ask Taber about how it worked on this Star Wars film, starting with the technical process behind taking the original plates and VFX shots and then bringing it through Stereo D’s conversion pipeline.

b&a: What is Stereo D’s process these days for stereo conversion?

Brian Taber: From a technical point of view, it all starts with receiving the footage and breaking that out. Every individual stereo layer and every individual layer that needs volume needs a matte. Then that gets re-composited in a depth budget area where we set our overall and internal volumes and layers for each individual object.

When the shift of pixels occurs, there’s the occlusion that’s uncovered. The compositing paint artists go in and recreate that pixel information. That gets re-composited altogether in both eyes. Our main proprietary software that does the stereo renders is known as VDX. Most of our tools sit inside of either Nuke or Maya or sometimes After Effects. We work with all VFX vendors, so all of our tools have to integrate with whatever information they’re sending over for us. That all drives through our proprietary VDX.

b&a: People often seem surprised that you really are pulling mattes and roto’ing so much of the frame for stereo conversion. This just seems like so much work. Can you tell me a little bit about the challenges with that and also whether you’re able to source mattes from visual effects vendors, say ILM in this case, and other vendors?

Brian Taber: The challenge is definitely breaking things apart in the frame. With Rise of Skywalker, J.J. shot a lot more practical sets, with large atmospherics and explosions. So there was a very talented team that would go through and roto and key all those individual mattes for our stereo team to work with. On most other big films these days, there’s a lot more CG effects. We work directly with ILM and other vendors to pull those CG elements and those individual compositing layers that would typically come with mattes and typically come with Z-depth. On Rise of Skywalker it was a much more brute force challenge. But it was well worth it in the end because we got to keep J.J.’s look and J.J.’s practical effects where they were used.

b&a: Is there a particular scene you can talk about in terms of the direction you got from J.J. and the other filmmakers for the stereo conversion?

Brian Taber: One of the first ones we showed him in sequence was the stormtrooper chase out in the desert. J.J. was really excited about that one. He was concerned with how quick cutting it was, but, working with ILM’s stereoscopic supervisor Sean MacKenzie, we got it into a good place where we could push the stereo but keep the eye drawn on the point of interest and the action. So that was J.J.’s concern to not give headaches but to play up this stereo where we could. But then, when we get into the dialogue with the actors, it was about making the audience not think about the stereo, just making them a part of the scene.

b&a: That’s a sequence which is largely CG backgrounds, but also with a lot of live action elements as well. At what point are you able to start working on the stereo for a scene like that?

Brian Taber: At the beginning of our process we’ll go in and grab all the drama plates, anything that doesn’t have any visual effects to it. And then we grab the next batch of plates that might have minimal visual effects and go on from there. And then, as temp visual effects start to come across, something that’s in the compositing phase – once animation’s locked – that’s when we’d like to grab those first temps and start working on them so we can start to get a sense of what the overall stereo is going to feel like and how shots are going to play against each other.

b&a: You mentioned that practical effects were a big part of this film. What are some of the very specific challenges you had to do when you’re converting explosions or debris or smoke or things of that nature? 

Brian Taber: One of the most challenging scenes from a practical standpoint was the film battle when Finn and Rose and others are on the side of the Star Destroyer. Most of that was a practical set with live explosions and smoke and transparency. Obviously there were set extensions and then the other ships in the background were CG, but a lot of that stuff around them was practical. Typically, like I said, we get layers from ILM and that comes broken out into sparks, fire, smoke – all separate with their own Z. That assists us to do the job more quickly, which allows ILM to make changes later down the line.

With this scene, with a lot of practical effects, it was a lot more of our roto and mattes team keying that stuff out or roto’ing individual pieces out and then separating out the sparks from the smoke as well as the fire and then separating out the fire from itself, because there’s multiple layers to the fire. Then once that’s broken out and given to our stereo teams, we’re back into our normal process.

b&a: How do you review shots in stereo?

Brian Taber: We have a process where our teams look at everything, originally on a large TV screen and then inside a theater. Then that gets sent to me as the supervisor and I watch everything in context to make sure everything’s going in the right direction, in line with what the filmmakers are looking for.

Once we have a good chunk of the scene in context that we feel confident that either the VFX supe Roger Guyett or the filmmakers can see, then we show them. We look at everything in context because sometimes it’s hard to see a one-off shot and understand the scope of the work and how it’s going to fit into the film. It helps us not be distracting from the story and lets us know where to push the stereo and where to incorporate the audience into the stereo.

b&a: What was your favorite stereo scene from the film?

Brian Taber: My favorite scene is the lightsaber battle on the water bridge of the wreckage of the Death Star. That one had a lot of cutting back and forth between wide shots as well as close-ups, as well as moving cameras. So that’s always a tricky challenge to not miniaturize the characters on the wide shots, and then give them enough overall stereo budget on the close-ups so that the characters feel like they have volume and they aren’t cards. That’s always a tricky challenge when you’re cutting back and forth.

Just given the epic-ness of that scene, we wanted to really get that correct. That was probably the one we spent the most time on just in individual adjustments of shots and convergence so that the audience wasn’t distracted, but they got to feel the overall epic-ness of that scene, with the crashing waves and cool lightsabers.