The insanely intricate work behind ‘Rise of Skywalker’s’ stereo conversion

Rise of Skywalker behind the scene

Here’s how Stereo D pulled it off

It still stuns me just how much effort is involved in the stereo conversion. It involves an analysis of the anticipated 3D experience for the audience, and then a whole lot of roto, matte creation and painting to fill in occlusion areas.

One of the latest big releases to receive the stereo conversion treatment was J.J. Abrams’ The Rise of Skywalker. The conversion was completed by Stereo D, led by stereographer Brian Taber.

I got the chance to ask Taber about how it worked on this Star Wars film, starting with the technical process behind taking the original plates and VFX shots and then bringing it through Stereo D’s conversion pipeline.

b&a: What is Stereo D’s process these days for stereo conversion?

Brian Taber: From a technical point of view, it all starts with receiving the footage and breaking that out. Every individual stereo layer and every individual layer that needs volume needs a matte. Then that gets re-composited in a depth budget area where we set our overall and internal volumes and layers for each individual object.

When the shift of pixels occurs, there’s the occlusion that’s uncovered. The compositing paint artists go in and recreate that pixel information. That gets re-composited altogether in both eyes. Our main proprietary software that does the stereo renders is known as VDX. Most of our tools sit inside of either Nuke or Maya or sometimes After Effects. We work with all VFX vendors, so all of our tools have to integrate with whatever information they’re sending over for us. That all drives through our proprietary VDX.

The original frame
The original frame.
Roto shapes
Roto shapes.
Depth mattes
Depth mattes.

b&a: People often seem surprised that you really are pulling mattes and roto’ing so much of the frame for stereo conversion. This just seems like so much work. Can you tell me a little bit about the challenges with that and also whether you’re able to source mattes from visual effects vendors, say ILM in this case, and other vendors?

Brian Taber: The challenge is definitely breaking things apart in the frame. With Rise of Skywalker, J.J. shot a lot more practical sets, with large atmospherics and explosions. So there was a very talented team that would go through and roto and key all those individual mattes for our stereo team to work with. On most other big films these days, there’s a lot more CG effects. We work directly with ILM and other vendors to pull those CG elements and those individual compositing layers that would typically come with mattes and typically come with Z-depth. On Rise of Skywalker it was a much more brute force challenge. But it was well worth it in the end because we got to keep J.J.’s look and J.J.’s practical effects where they were used.

b&a: Is there a particular scene you can talk about in terms of the direction you got from J.J. and the other filmmakers for the stereo conversion?

Brian Taber: One of the first ones we showed him in sequence was the stormtrooper chase out in the desert. J.J. was really excited about that one. He was concerned with how quick cutting it was, but, working with ILM’s stereoscopic supervisor Sean MacKenzie, we got it into a good place where we could push the stereo but keep the eye drawn on the point of interest and the action. So that was J.J.’s concern to not give headaches but to play up this stereo where we could. But then, when we get into the dialogue with the actors, it was about making the audience not think about the stereo, just making them a part of the scene.

A final shot from the stormtrooper chase sequence
A final shot from the stormtrooper chase sequence.

b&a: That’s a sequence which is largely CG backgrounds, but also with a lot of live action elements as well. At what point are you able to start working on the stereo for a scene like that?

Brian Taber: At the beginning of our process we’ll go in and grab all the drama plates, anything that doesn’t have any visual effects to it. And then we grab the next batch of plates that might have minimal visual effects and go on from there. And then, as temp visual effects start to come across, something that’s in the compositing phase – once animation’s locked – that’s when we’d like to grab those first temps and start working on them so we can start to get a sense of what the overall stereo is going to feel like and how shots are going to play against each other.

b&a: You mentioned that practical effects were a big part of this film. What are some of the very specific challenges you had to do when you’re converting explosions or debris or smoke or things of that nature? 

Brian Taber: One of the most challenging scenes from a practical standpoint was the film battle when Finn and Rose and others are on the side of the Star Destroyer. Most of that was a practical set with live explosions and smoke and transparency. Obviously there were set extensions and then the other ships in the background were CG, but a lot of that stuff around them was practical. Typically, like I said, we get layers from ILM and that comes broken out into sparks, fire, smoke – all separate with their own Z. That assists us to do the job more quickly, which allows ILM to make changes later down the line.

With this scene, with a lot of practical effects, it was a lot more of our roto and mattes team keying that stuff out or roto’ing individual pieces out and then separating out the sparks from the smoke as well as the fire and then separating out the fire from itself, because there’s multiple layers to the fire. Then once that’s broken out and given to our stereo teams, we’re back into our normal process.

A shot from that Star Destroyer sequence
A shot from that Star Destroyer sequence.

b&a: How do you review shots in stereo?

Brian Taber: We have a process where our teams look at everything, originally on a large TV screen and then inside a theater. Then that gets sent to me as the supervisor and I watch everything in context to make sure everything’s going in the right direction, in line with what the filmmakers are looking for.

Once we have a good chunk of the scene in context that we feel confident that either the VFX supe Roger Guyett or the filmmakers can see, then we show them. We look at everything in context because sometimes it’s hard to see a one-off shot and understand the scope of the work and how it’s going to fit into the film. It helps us not be distracting from the story and lets us know where to push the stereo and where to incorporate the audience into the stereo.

b&a: What was your favorite stereo scene from the film?

Brian Taber: My favorite scene is the lightsaber battle on the water bridge of the wreckage of the Death Star. That one had a lot of cutting back and forth between wide shots as well as close-ups, as well as moving cameras. So that’s always a tricky challenge to not miniaturize the characters on the wide shots, and then give them enough overall stereo budget on the close-ups so that the characters feel like they have volume and they aren’t cards. That’s always a tricky challenge when you’re cutting back and forth.

Just given the epic-ness of that scene, we wanted to really get that correct. That was probably the one we spent the most time on just in individual adjustments of shots and convergence so that the audience wasn’t distracted, but they got to feel the overall epic-ness of that scene, with the crashing waves and cool lightsabers.

4 Replies to “The insanely intricate work behind ‘Rise of Skywalker’s’ stereo conversion

  1. Nice Article.

    There were some stellar shots throughout to be sure.

    But with basic errors throughout like background objects in foreground space and constant depth issues with panning and dolly shots, it hammers home just how much better filming in native stereo is. And at most, even the best conversion will only be 2.8D

    1. I disagree about native vs conversion. So many of the native 3D films from the contemporary era were shot VERY conservatively to make stereo shooting easier. Tron, Dredd, Planet of the Apes, the Hobbit movies all have such shallow depth there’s almost no point to being in 3D. That’s what killed the format, people paying extra for 3D and barely even being able to see the difference.

      Once conversions took over and we finally started to get good depth people had given up on it. Movies like Rise of Skywalker and nearly all the Marvel movies are excellent conversions and have all the 3D excitement of any native shot movie.

      Too many people equate native stereo with automatically being good but that’s a load of BS. I’d rather watch an imperfect conversion that has good depth that a native film shot with an inch of interaxial.

      1. I just recently watched 4 of the eight films you mentioned above having shallow depth… and I am truly sorry if you have had bad experiences with them in 3D, but at home with my laser projector and my humble 135″ screen, The Hobbit has stellar 3D and Dredd is just gorgeous.

        I agree with Tron, as I heard Kosinski wanted to push it, but Disney wouldn’t let him.

        As for Marvel films, most of their spectacular CG work especially battle sequences and such are completed in native stereo with most dialogue sequences and such being the parts that are converted. But a dark secret of native stereo films is that in the end quite a lot of that has to be converted anyways, but they at least have the second camera info to do a decent job of it.

        A great example of how close, but not quite, conversion is to native stereo is Hansel and Gretel: Witch Hunters. Started filming flat, halfway through changed to stereo, as they didn’t film it in order some of the conversion shots are interweaved with the native.

        I don’t equate native stereo with automatically being good, as there are some cinematographers/directors out there who just don’t get stereo. But I would go a film shot in native stereo with an inch or two of interaxial over a conversion that has depth but lacks dimensionality.

  2. Interesting there is no mention of Silhouette. I guess the roto part is too “boring”?

Leave a Reply

%d bloggers like this: