Decals, practical glass hits, FX sims, meticulous compositing, and more.
A key moment in Cary Fukunaga’s No Time To Die sees Bond (Daniel Craig) and Madeleine (Léa Seydoux) under a barrage of bullet fire in an Aston Martin DB5 in the middle of a square in the Italian town of Matera. That sequence involved a careful coordination of special effects and stunt work, including the bullet hits, which was then augmented by visual effects from Industrial Light & Magic.
Here, ILM CG supervisor Stephen Ellis tells befores & afters about taking what were sometimes ‘decals’ of cracked glass bullet hit shapes on the DB5 on set, and then crafting final shots using several different methods, all while maintaining bullet hit continuity. Ellis also discusses other VFX work by ILM in Matera, and one unusual challenge which came during tracking super-high resolution IMAX film plates for chase sequences.
b&a: I really liked the mix of approaches to achieving shots in the film, from stunts to special effects and digital effects and digital environments. So, congratulations on being part of that work.
Stephen Ellis: It was a really good example of where you are partnered with the Production’s other departments. You’re right, loads of stunts—undeniably amazing stunts—and loads of practical effects. But the visual effects team was there at the end to make sure that we could pull all of this together. So, yes, we had stuntmen jumping off bridges, but we also had digi-doubles jumping off bridges as well.
The way it was edited, we tried to use as much of those remarkable stunts as we could, but we then came in to supplement. Maybe there was an angle that they didn’t quite capture that the director wanted. It’s very fast paced, so a lot of our shots slot into this high pace, high action, very “frenetic” sequence. I think when you look back at the body of work, you’d be quite pushed to see any of the shots that actually stood out as visual effects shots. I’ve worked on a couple of Bond films, and it’s always about creating effects that are seamless to the viewer, which as a visual effects artist is absolutely what I strive for. I’m really passionate about those really truly invisible effects.
b&a: Can you talk first about the environmental work in Matera?
Stephen Ellis: We did quite a lot of environment augmentation. There was a bridge at a site nearby that we did a lot of photogrammetry on. Then we took our plates from Matera and we bolted in this VFX bridge, which allowed us to make much nicer compositions or to be able to design the kinds of compositions the director was after, where we could use the bridge to draw your eye into the focal point of the shot which was the town.
There’s one shot, where the DB5 is going through a tunnel approaching Matera, there are so many visual effects in that shot; lots of plate stitches, digital environments, and for the town itself, there were lots of augmentations added there also. The creative story point is that Bond and Madeleine arrive at Matera during a night of celebrations where people around the city are burning their wishes. There were lots of digital crowds added and then lots of digital fires and embers, I think we really helped bring it to life. We even replaced the DB5 in that shot just because it was coming through a very dark tunnel. We were given beautiful IMAX plates to work with.
b&a: You mentioned photogrammetry of the bridge. Did ILM get to be there to take reference and gather all the photo and Lidar reference?
Stephen Ellis: Yes, ILM visual effects supervisor Mark Bakowski was on the shoot. The production themselves had a big team of data wranglers so we would typically get the data from them. It’s a funny thing to say, but sometimes we’d receive a plate with what was considered these slightly ‘pedestrian’ backgrounds of these beautiful Italian rolling hills. It wasn’t quite what was required for the script, so we replaced the rolling hills with the facades of this incredible town of Matera with all the caves and the amazing way that the buildings are built into the rock. A lot of the team from ILM were from our generalist environments department and they really did some classic digimatte work on this.
b&a: Was there a particular approach here for that sort of work?
Stephen Ellis: We tend to use whatever approach works. I used to work in the digimatte department. My preferred approach is always to try and start with photography. Particularly on a Bond film, you want it to look like that location, so of course you start with the location photography. That’s what I mean about this classic digimatte work, a lot of the approaches that the generalist team were doing was taking photography, stitching it together to create backgrounds and then building into it. Depending on shot complexity we’d typically be then making the photography two-and-a-half-D, and then combining that with renders of trees and forests—which are not going to be done as images on cards—because you’re not going to get the right result.
There’s one little beat where Bond jumps off the bridge with the rope; some of the shots looking back into the ravine are fully digital renders where we started with photogrammetry backgrounds with digital matte paint finishing. But then the trees are so close that those trees are added as digital 3d renders.
b&a: And you mentioned something which I really liked, which is the way it’s edited, there’ll be an ILM visual effects shot, and then it’s cut right up against a real one. One of the ones I’m thinking of is when Bond has to hide behind that stone so that the car goes over him. I mean, I assume it’s a CG car.
Stephen Ellis: It is CG, you’re right. I mean, cast and crew safety is paramount, you would never put an actor in that kind of danger. So yes, CG car for the bump into the stone and the dive over, but cut around practical elements of the same car. I really do want to give a bit of a nod to the lookdev team because those renders are really pretty seamless. It was such a good match to the practical car.
That scene starts, at the head of the cut is a car at the far end of the bridge. That’s a real car, and that’s your practical reference there, although when Primo turns up on the bike, we were doing digital head replacements. We put the digital car shot in and then the reverse shot would be a practical car again.
b&a: What about that very keystone motorcycle jump?
Stephen Ellis: It was a really nice mix between visual effects and the other production departments. It was quite remarkable. They did that stunt for real. They built this amazingly scary ramp that was painted blue. That stunt was actually shot practically and we didn’t change any of that. The main bulk of our work was actually just taking out the blue ramp and making it look like old brick and rock.
We also did a head replacement because the stunt performer was wearing a safety helmet for obvious reasons. The bike sequences were really interesting. Practically all of those shots are head replacements combined with digital doubles for Daniel Craig.
I think right until the last shot where he steps off the bike, I think even the step off includes at least portions of our digi-double. We’re not fully digital, but we would keep maybe parts of the clothes, and add digital trousers and shoes, because the stunt performers had special effects safe footwear on. We would just do little augmentations, while trying to maintain the performance of the stunt performer, and then it would be a digi Bond head. It cuts so quickly into the then “practical” Daniel Craig actually walking into the hotel. It’s a really nice bringing together of all of the techniques that we used to shoot it. The sequence makes perfect use of invisible visual effects to augment the incredible practical stunt work that Bond films are known for.
b&a: How did ILM approach digital Daniel Craig shots?
Stephen Ellis: ILM was the lead actually on the CG Daniel Craig, and several other digi-doubles. One thing you really might not know, and I’m quite proud to be involved in this project because of this, there was a sequence inside the boat where Ash and Bond have a fight. That was captured with two talented stunt performers. So that sequence required face replacements. I think it’s pretty seamless work when you look back at it.
b&a: That’s amazing.
Stephen Ellis: The director, Cary Fukunaga, wanted to shoot a facial performance for every shot. He didn’t want a team of animators doing keyframe facial expressions; he really wanted the performances coming from the talent themselves. We had an ILM Medusa and Anyma shoot for every scene, which meant we could rely on that facial capture.
Now, if you look at Daniel Craig’s acting, the character that he’s playing, it’s quite a cool, calm –
b&a: – stoic?
Stephen Ellis: Yes, stoic performance. We’ve got quite a lot of experience in doing this here, so we would show the first version of the performance to see how it was working for the shot. If the request came through for a facial performance change, we could take performances from other takes and mix them in to get the best result for the shot, but the approach was always face cap; Medusa, performance driven.
b&a: And does that just involve setting up a Medusa and Anyma rig?
Stephen Ellis: Yes. The production and Charlie engaged with that. We had a list of all of the shots that we wanted to do facial performances for. We would have Daniel in the right shooting conditions to get the best performance and to get the best animated mesh back, so we control the conditions pretty tightly. We would get Daniel or Nomi and all of these characters in and they would shoot the performances that the director was after.
b&a: Let’s turn to the Aston Martin DB5 work, maybe concentrating on the donut square sequence, which I thought was amazing. Clearly Chris Corbould, and the Stunt Coordinator, and special effects work there is incredible. But there’s some really nice augmentation from ILM. What was that?
Stephen Ellis: As you might imagine, the smoke coming out of the exhaust involved heavy effects sims that would augment the practical effects. On-set in Matera, the special effects department did set up squibs and practical elements that would fire off around the environment when the DB5’s rail guns were firing. We took all of that and added to it. We cleaned up where SFX rigs may have been visible, but we’re always just trying to remain faithful to what was shot. The thing that I’m most proud of and I think people probably won’t realise that we did is that the augmentation of the glass bullet hits; building upon the guide from the special effects team who put decals of where the bullet hits would be on the DB5’s glass.
b&a: Okay, wait. This is one of my favorite things. The bullet hits. My next question was going to be, I am so impressed by the continuity of the bullet hits, and I just don’t know how something like that is achieved.
Stephen Ellis: That took a lot of work. There were five DB5s that were actually built that were taken to Matera. The special effects team put these decals onto the DB5s and they shot the sequence. Of course they’re not going to be firing real guns at safety glass particularly when you’ve got stunt performers inside. The brief came to ILM to expand or augment these glass cracks.
One thing that’s really interesting looking back on this sequence is just how many techniques we employed. If it was a wide shot or a close-in, it could have been a mix of 2D or fully 3D rendered glass, or composited photographic elements, depending on which car they used and what the shot requirements were.
The way that we approached it was, we had the decals from the special effects team – so that’s what was shot. In comp, we kind of started at the end of the pipeline and went backwards. So the compositing team took the track of the DB5 and baked down the position of the decals into the UV space. It went back up the pipe to the FX team who could then take the 2D projected decals, then take that image input and with various tools we developed within Houdini, they could trace those 2D locations back into a 3D representation of where the decals were and how they were moving in 3d space.
One of our artists, Michael Cashmore, came up with this beautiful system that could work with either this 2D input, or a 3D input coming from the layout team’s tracking locators. One of the problems that we had was, we only had one digital DB5, but there were five different stunt cars that were taken to set. They might be manufactured slightly differently, you couldn’t tell by looking at them, but when you really got into the fidelity of the tracking that we do, you start to notice, this car is a minutely bit longer or the windscreen, and a bit different on this one, etc.
We’d have our layout team put 3d locators where each of these decals were and track these. Then that could be cached to the FX team as a moving bundle of 3D points. Michael set up this technique where he could read in either the 2D ones or the 3D ones and trace their locations back into the DB5 model, which would then drive a next-generation glass fracturing process. This would normally be based off of photography, so we’d give the FX team the photographic elements of cracks that we wanted to use. And then for some of the shots the compositing team were taking the elements and directly mapping them onto the car model.
The FX team were taking the same set of images and shattering the glass based off of the same crack patterns. But then some of the shots got way more complex. You had people firing at the cars and you actually had simulated glass coming off. There’s one amazing shot where it’s really close up on the inside and it’s looking at that pattern, where Primo’s firing at the foreground window. That was actually an element shoot, but we augmented it, putting digital Matera in the background.
The continuity was something that we really paid close attention to. Mark Bakowski was really eagle-eyed on that, making sure that all of those glass hits did carry through the sequence.
And then right at the end, in one of the last two shots, you’ll see it where the car’s speeding towards the train station. It was one of those shots which was actually flopped when we got the plate. So we had to flop the plate, re-flop all of the signage, because of course that would all be backwards. And then the DB5 and the characters inside needing to be flopped, so all of the continuity of the bullet hits was the other way around. So we had to replace the DB5 in the plate with a fully digital car on a flopped plate.
b&a: It’s amazing work.
Stephen Ellis: I’m really proud of that glass work, as well as the sequence overall. The system the team built was really robust to be able to get an input from comp or an input from layout and produce the fracturing. I really need to give a nod to the comp team though because they were the first team to start driving and developing the look of this glass bullet hit effect, as well as the sequence continuity. As the shots developed, comp then started raising the issue of which elements they were going to need to really sell the effect. That’s when we started rebuilding our systems to try and help give them elements that they would need. And then by the end of it, we had shots where it was just a full digital car with digital glass, with digi doubles inside as well.
b&a: And just in that ‘donut’ square, I imagine you’re just doing other fixes like tire marks and things like that.
Stephen Ellis: It was remarkable what our Paint department was doing. If you can imagine, the process of filming take after take builds up all this burnt rubber on the street. We had to replace the ground back to the clean streets on practically every car shot. In fact, it’s not even just on the square; for the whole sequence in general, all of the cars chasing around the city; some of those cobbled streets were just covered in the previous takes’ tire marks, so most of the cobbled streets are replaced.
There are quite a few fully digital shots as well within that sequence where the filmmakers ultimately wanted shots that they didn’t capture with the main cameras. The production had an array vehicle that we would get a lot of footage from. So for some of the shots we could take that array footage and stitch it together to create new plates. Sometimes it was entirely digimatte. So we’d have LIDAR, we’d create digital models of the environment, but we’d take still or moving photography from the set. We’d project it all out and we’d just make up shots; the shot where the Land Rover smashes into the DB5, that’s a completely digital background and a digital Land Rover as well coming in, so it’s a foreground plate of Bond and Madeleine, but then everything out the back is digital.
b&a: Those array vehicles are kind of cool. What do you actually get from them? Is it literally six pieces of running footage?
Stephen Ellis: Yeah, pretty much. Typically, it would be shot with RED cameras, since they’re high resolution and small and you can sync them very easily. We also had a fisheye camera on top, and you can transform that into a high dynamic range IBL to render with if needed.
b&a: You already mentioned the smoke when he does the donuts. Were you having to integrate any kind of special effects smoke with your CG smoke or did they just shoot some great clean plates for you?
Stephen Ellis: No, it was additional FX integrated with special effects. I mean, as visual effects we would always say, please shoot something. Even if we end up using this only as reference, it gives us something real there that we can latch onto. I think that’s paramount to the success of the effects work in the Bond films.
b&a: The other thing I wanted to ask you was about integrating ILM’s digital work into the plates. What were the challenges there?
Stephen Ellis: One particular challenge actually in Matera was, it was shot on IMAX, so we were getting IMAX plates. Those IMAX cameras have an absolutely massive film back, something like seven centimetres. And those cars are driving over cobbled streets as well. When we were trying to do the tracking work, we’ve never really experienced that level of vibration. You’ve got the suspension of the car shaking around and you’ve got this huge camera, the massive film back, and you’ve got the film in there that’s actually shaking around, too.
Our layout team did such a good job tracking those plates. All of our tracking points were locked on to all of the 2D tracks and everything was tracking really well. But we were occasionally seeing slightly different motion blur on our renders compared to certain parts of the plate. We internally talked about it and we concluded that it was almost like a difference in sub-frame curves because there was so much vibration. So we started considering, should we even be doing subframe tracking here?
I mean, we wouldn’t have had any frames of course to be doing sub-frame tracking with, but we realised that even between frames – the film or the film back–whatever it was, was vibrating in a way that certain parts of the plate were getting a kind of extra blur.
When you’re working at that really high IMAX resolution, there were times when we started evaluating each quadrant of the image and we provided multiple QC views centred around individual tracks for different parts of the image. So everything we were rendering, we were also having to occasionally go through each quadrant of the frame and work out; everything is sticking but is the changing blur working correctly or do we need this kind of sub frame, extra high frequency blur? So the comp team were then occasionally adding these additional vibrational blurs across portions of the frame on top of what we were rendering to get that true integration. It’s really mad when you think about it.
b&a: Wow, yeah.
Stephen Ellis: If you think about the minutiae of those curves, the fidelity that we needed to be working at and the fidelity of how we were reviewing everything as well. It was the last project I worked on before COVID, where we were reviewing everything still massive on the screen in a review room.
b&a: How did all that impact the way you ultimately completed the shots?
Stephen Ellis: It was a combination of teams. The tracks were for all intents and purposes perfect to the data that we had, which was obviously shooting at 24 frames per second. You could step through each frame and you could see that all of the tracks were sticking perfectly, but it was the way the camera curves would be changing between frames that we needed to be matching. We were often giving comp a cropped-in section of a certain corner of the plate and rendering them an image that was centred around a specific track point, with the plate moving around that track so that comp could see this isolated vibration, which incredibly they managed to match to.
b&a: That’s fascinating. I wonder if other productions have come across that, but it could be very particular to shooting a car chase on cobblestones and on IMAX.
Stephen Ellis: I think that’s certainly the most challenging plate shoot that I think I’ve seen our layout team deal with. Number one’s the resolution, those huge plates, but number two is that it’s a car chase over cobblestones where the camera team is being shaken all over the place. We really were having to think about, okay, now we’ve got all the movement of the car locked down, what else do we need to match? It was a really scientific approach from lots of different people to figure out how we successfully did this.
