One shot even saw Digital Domain replace a real car with a CG one going through real water barrels.
For scenes of Spider-Man facing off against Doc Ock on a freeway overpass early on in Spider-Man: No Way Home, production shot plates on an Atlanta backlot. Digital Domain then took those as a starting point, building out a huge environment, added CG vehicles and sometimes CG characters, and generally crafting an additional sense of destruction and mayhem.
Here, Digital Domain visual effects supervisor Scott Edelstein tells befores & afters how it was done, including tackling things such as speed ramps, a digital Doc Ock, and even making changes to the sequence after the first trailer was released following some fan comments.
Building the freeway environment
b&a: One of the interesting things about your work for the freeway and overpass sequence is the environment extensions. Basically, you had to build a whole world there, right?
Scott Edelstein: Yeah, absolutely. I mean, one of the things that we’ve learned over the many years of working on Marvel films is that as the scope evolves throughout the editorial process.They change things to make the movie better, so we have to be ready and able to put a CG camera into a CG environment, anywhere, looking in any direction. For a sequence this large, we knew that would be the case from the start, so we spent as much time as possible building out our CG environment to hold up.
For the bridge sequence the production did shoot some plates in New York. We ended up using some of them, but most of the plates were shot on stage in Atlanta. There was a lot of reference photography from NYC though, and they tried to shoot plates as much as they could. They also had a helicopter, and shot a bunch of aerial plates for reference for us as well.
We ended up taking some of that, and some reference we found from the area, including Google images. We used anything we could find. We put together a photographic environment of those images that we could reference, and then lined up the LiDAR from the actual bridge. We then just iterated for months, over and over again on everything in CG, just building all the assets we needed. We’d render it and then compare it to all the different angles from that environment until we had as close to a one-to-one match as possible.
Daylight isn’t exactly an easy lighting environment for such a huge undertaking, but the good news was that the time of day was consistent. Once we had it matched and we had the environment built, we could then just put the camera wherever we wanted and give them any shot they wanted.
Principal photography took place on a backlot in Atlanta, so we started with the plates they shot there. What often ends up happening is the camera direction changes, the story changes or the edit changes. And then suddenly the cars should be the other direction, or Spider-Man should be in a different part of the freeway.
For this film, a lot of it ended up just being all CG, or we replaced quite a bit of it. With Alfred Molina, for example, he’s real for a lot of it, or at least his face is. But the lighting direction would end up changing, so we’d have to replace his hair and do a lot of comp work to make the lighting direction work. A lot of times his glasses were CG for the reflections.
b&a: It’s interesting, because what you’re saying is that just by doing all that work, you provide a lot of flexibility to the filmmakers effectively, for what is a huge action scene. I think that’s something that’s not always discussed in visual effects. You’re not just filling out the frame with what wasn’t shot. You’re actually helping to achieve a new, not a new sequence, but a fuller sequence.
Scott Edelstein: Yeah, exactly. That’s a good way to put it. It’s not different, but it is enhancing the story, or allowing them to add shots and do things slightly differently than they originally did. Even if they put it all together with what they shot, a lot of times they come back and they’ll say, ‘Oh man, I wish this camera was a lock off.’ Or, ‘I wish it was moving just a little bit.’ Or, ‘this is a really exciting shot, but it could be way cooler if…’. We’re there to fill in all those ‘ifs.’
CG cars
b&a: You mentioned building everything out, but there are cars or remnants of cars on the practical sets. What are you doing to photogrammetry or scan or replicate those? And then how are you also building out so many other cars? Is it a library that DD has? Or are they brand new models?
Scott Edelstein: If it was only that easy… At the beginning of every show we think, ‘It would be great if we just had a library of cars, or NYC buildings,’ etc. etc. But at the end of the day, things progress so quickly. Yeah, we have lots of cars and buildings in a library, but now we need cars that match the ones that are already on the bridge, and match the era.
For Spider-Man, they did have practical cars on set, and we built a bunch of vehicles that matched what they shot. So if we did use a plate, we could tie back into that. The production scanned and completed texture shoots for every single car, so we had reference materials to rebuild them. We then based the layout on the immediate area where a lot of action happens.
But as productions tend to go, vehicles don’t generally stay in the same place, and Spider-Man was no exception. The production moved the cars around and changed their position from time to time, so continuity isn’t something that you can always match perfectly. Instead, we make each shot flow together as best as possible.
As far as the rest of the cars in the city, we built around 50 cars that are high enough resolution that they stand up and have tires that turn. The primary goal is just to ensure that it doesn’t look like cars are floating around out there. Then you vary them with color and shaders. So you might have 24 Tauruses driving around, but there’s a blue one and a red one and white one and so on.
CG destruction
b&a: Let’s talk about destruction for a minute, other than what the characters are actually doing to the cars. I mean, sometimes the cars get hit by the characters. Sometimes it’s by, like, a concrete pipe.
Scott Edelstein: We had a lot of good references. The special effects supervisor Dan Sudick did some really cool practical effects that we could reference for all that stuff. He actually dropped a pipe on a car at one point, and pulled a car through a hole in the ground. He bent cars in half. We would call them ‘taco’ cars. We had lots of good references for what that would look like.
Then you have to figure out how many cars you need to interact with because those cars have to be built to a higher level higher resolution and be able to be damaged by the effects team. And then we had to blow cars up. Obviously, Doc Ock picks up a lot of cars and throws them. They’re getting pushed through the concrete barriers, or falling off the side of the bridge, or getting hit by things. And then you have to track all that damage through the rest of the show, the rest of the sequence.
So if one specific car is damaged in a shot, now it’s always damaged. So there’s a team of layout people paying attention to that kind of damage continuity, updating shots and making sure that everything flows through the rest of the sequence. So once things are all established, it’s a bit more automated, but there’s a lot of manual work that goes into figuring it all out along the way.
The water barrier was real water…
b&a: There’s even that shot of a vehicle going through the water protection barrier. This scene is like a movie in itself. I’m actually really interested in a shot like that. Are you tending to give that to one specific Houdini artist?
Scott Edelstein: That’s the shot where Doc Ock picks up the car and throws it at Spider-Man. What was cool is that’s one of the practical effects that we used. They actually did throw a car through those water barrels, and all that water is practical.
b&a: Oh, wow.
Scott Edelstein: It was super cool. What we didn’t use was the actual car that they threw into the water. So as fun as that sounds, we had to remove the practical car from the water because the angle and the type of car needed to change. The animation had to match the previous shots, even though the practical car didn’t. Still, it was great to have that element because it was super cool, and it ends up looking very realistic.
For this particular shot, consistency is key, so we couldn’t mix and match team members. We had to put CG cars around the practical water, and then have it interact with each vehicle. There was one FX artist, one compositor, one lighter and that kind of thing. For a shot like that, where you’re trying to use the practical element, the FX artist provides additional CG elements to help the compositing team blend all that practical water into the CG. Because, the background behind all the water is CG, the road surface and most of the cars around it are also CG, and the car going into the water is CG. The only thing real is the water, as funny as that sounds.
Speed ramps in VFX
b&a: It’s amazing work. Seeing as we are talking about that particular shot, and it’s where the car is thrown at Spidey, and he leaps over it, one thing I love about that, unless I’m totally wrong, is a speed ramp change. I’m always curious in visual effects, how tricky that is?
Scott Edelstein: Even if there’s not, it can be hard. It really depends on how they’ve done the speed ramp, if it’s linear or if we have to re-create frames. But a lot of times, what you end up doing is just a lot of clean-up on the re-time. So you go through Nuke, and you feed it through a Kronos-type tool, which is the way that Nuke re-times it.
If it’s just dropping frames, then it’s great; it’s not much work. But if you’re interpolating frames and coming up with new content, then a lot of times there’s artefacts, lots of artefacts, that then have to go and be painted out by somebody. So what we try to do is get them to lock down what the re-time is going to be pretty early.
We’ll go back and forth with our editorial and their editorial to try and come up with what the re-time is, because once we’ve established that and start working, it’s a lot of work. So once you start down that road, you want to be locked.
Then typically what we’ll have is integration; the team that tracks the cameras, if there is a plate, they’ll actually do that to the non re-timed version. Because re-timing a camera or a body track for a person or something like that is relatively simple. You basically apply the same curve that you are applying to the image to the motion curve of the camera.
If they do then change something, you don’t have to re-track it over and over again. But you end up doing roto to the re-timed image, so that you don’t get roto artefacts like you would if you did try and re-time the roto. So that can be kind of painful when things change. When it is a ramp, that’s the harder part, because you have to feed that ramp into things like Houdini for the particles or Maya for the animation, or both. And a lot of times, it’s iteration. Because even if you apply the same speed ramp that the image is getting, it doesn’t look right when you give it a cloud of smoke or a water explosion.
When there is a person or a plate involved, it can be a little more complicated to get those things to play nicely. It’s easier with all CG. But even when it’s all CG, you’re talking about a speed ramp that’s being applied in Maya and V-Ray, and then also being applied in Houdini or Solaris and Redshift. And sometimes just matching that speed ramp between multiple packages can be difficult.
Doc Ock
b&a: I did want to ask you about Doc Ock, mostly because although this is a heavy CG sequence, I honestly felt the arrival of Doc Ock and his destruction and movement, there was a part of it that felt very much like they could shoot it.
Scott Edelstein: Right, I mean, the good thing about shooting plates, even if we don’t use all of it one to one, is that we base our cameras and character animation on what’s been shot. A lot of times, we’re using Doc Ock’s head at the very least, even if we’re replacing his body. We’ll still base everything that’s happening on the physical camera motion and lens that they used to shoot the plate.
And if we don’t, if we’re doing a completely CG shot, we still use the same lens package. We tie ourselves to whatever lenses they actually used on set, and we’ll try and pick lenses that fit that type of shot. For example, if we know that the director likes to shoot specific types of shots with a 40 mm lens, we’ll choose that.
And then we really try to be conscious of making cameras in CG that create shots that could be replicated physically. I think one of the things that ends up making shots look very CG is when cameras are doing things that are impossible. It still looks cool and all your CG can look very real, but if the camera is just doing a crazy thing that could never happen, your brain just flips a switch and it changes the way the image is perceived.
Even with Doc Ock on those arms, we talked really early on with Kelly Port, Jon and our animation supervisor Frankie Stellato, about making sure that there was always the sense of weight transfer between the arms, and that there was always balance. He could never just be standing on one arm with the other three arms doing weird things. He always had to be stable in real life. I think that lent a lot to making it feel better and making it feel more real.
Similarly, a lot of times, you animate to camera and whatever’s happening behind camera is not necessarily physically correct. But Doc Ock always had at least two feet or tentacles on the ground. And if he was picking up a car, he was always putting his weight in a way that would make sense. I think all of that lends to the realism of the animation, and then also just to the scene itself. It doesn’t look like something that could never happen.
b&a: What was the approach to getting a digital Alfred Molina? I mean, DD does so many digital characters, but how was it approached here?
Scott Edelstein: At this point, digital doubles are used often enough that capture is pretty standardized. We get a high res scan and textures of the body, costume and face, along with a FACS session of the actor so we can match their facial expressions properly.
We then took his clothes and shot them as flat as possible to get swatches of the texture. It then went through the modeling and texture look-dev pipeline. CFX also has to build a rig for his clothes, and check how the skin is going to react in the different environments.
For Doc Ock’s tentacle arms, there was a practical model that Sony had on display that we roughly based our model on, and production also built a practical model to use as reference on set. That gave us another example to help understand how the director wanted the arms to look in his shots. From there, the model or textures and look development evolve throughout the shoot, and you end up with an updated version that is a mix of old and new.
There’s also a moment in the film where the nanotech in Spider-Man’s suit transfers onto Doc Ocks’s tentacles, and figuring out how that gets applied was another challenge. That became an iterative process, just like figuring out how the tentacles worked. One of the biggest challenges with the arms was that you don’t want them to feel like they scale, or get longer and shorter in the shot while he’s doing something. So if he’s picking up a car and throwing it, you don’t want it to feel like he’s creating more vertebrae or sections of the arms as he’s doing those actions. We had a system that allowed us to make the arms longer and shorter for different shots, but we wouldn’t have them grow in the middle of a shot.
b&a: There’s obviously great reference from the previous films, and then you’re taking it further. When you or Kelly or the director are reviewing the tentacles shots, are you doing side by sides with the original film?
Scott Edelstein: For sure, we actually ended up with a library of reference from Spider-Man 2 we could review for different actions and find a set of rules for how the arms behaved. What colors are the lights and why? When do they change? How does Doc interact with each arm, and what are they able to do? We learned that Alfred Molina had named each arm individually, on the set of the original film, and each one has a distinct personality that he could play off when he interacted with them. He also named each arm: Moe, Flo, Harry and Larry, and we kept those names. We’d give and receive notes internally like, “can Flo move a little faster,” or “can Larry hit a bit harder?”
Obviously, technology has changed since they did that film. So that’s where you get a little bit of the stuff looking slightly better or fitting in better. We definitely tried to be as true to the previous films as possible while still updating it, and making it look even cooler for this new audience that we’re presenting this to.
Fan feedback
b&a: I was just wondering if as a VFX sup, you ever get worried, or that’s not even the right word, of a fan doing a comparison video. I mean, maybe these are out there already.
Scott Edelstein: I think it’s fun to read the anticipation, the fan anticipation and their ideas of what it’s going to be or why. When the arms started, the pictures of the arms got out there with the nanotech on it, and everybody’s like, ‘Ooh, I think it’s this,’ or, ‘Oh, it’s that,’ or, ‘He’s taking Stark tech and making it his own, and he’s going to be so much more powerful.’
The whole time we know what’s really happening, but in terms of the fan theories, I mean, I think it’s cool. I think it’s fun when fans spend that much time, because obviously we are spending that much time putting that work in to get those details right. And if somebody’s going to go break it down like that, then that’s super cool.
b&a: I mean, I even got sent this thing from my nephew yesterday, but it’s Doc Ock hovering and I’m not sure if his arms are in the frame or not. And a lot of people are making funny memes about it in terms of like, the character being set to zero in Maya, or in a game. Have you seen this?
Scott Edelstein: I haven’t. That’s so funny. I mean, it is an odd character — he is floating around. And a lot of the bluescreen plates of him are just him on wires, floating there. We had to replace his body most of the time, just because of how the physics worked there.
