The VFX studio also meticulously modeled a CG replica of the British Museum atrium.
One of the sequences shrouded in secrecy in Sam Raimi’s Doctor Strange in the Multiverse of Madness saw Wanda Maximoff (Elizabeth Olsen) take down most of the members of Earth-838’s Illuminati–Captain Carter, Captain Marvel, Black Bolt and Reed Richards–in a dynamic super-powers and magic-filled battle.
Industrial Light & Magic handled the visual effects for this battle, which takes place in an atrium that drew extensively from the real atrium of the British Museum. Here, ILM visual effects supervisor Julian Foddy, who worked closely with production VFX supervisor Janek Sirrs, breaks down for befores & afters the key aspects of the Illuminati fight, including the atrium build, effects sim work, and the approach the VFX studio took to doing stunt face replacements using a specialized face swap and photo-reference workflow.
Building the Illuminati atrium
Julian Foddy: The original plan was to shoot in the British Museum. However, when you realize that if you’re going to set pyrotechnic effects off on the floor, then you’re going to need to have a raised floor deck. And if you’re going to smash walls up, you’re going to have to have either fake walls, or that’s all going to have to be CG. And if you’re going to have people flying around and doing stunts, you’re going to need to do wire work, which means you’re going to have to have scaffolding, etc., for the wires. So we soon realized that by the time we’d done all of that, there would be none of the actual British Museum atrium left visible.
The decision was made, instead, to build a greenscreen stage at Longcross Studios that was going to represent the atrium. The slight advantage of the COVID lockdown was that we were basically given unfettered access to the British Museum during the daytime to go and shoot and scan. They could close it off to get decent, clean shots because the place was deserted. A scanning team was basically given 24 hours to do whatever they wanted. They came away with a fabulous set of data–an incredible, highly detailed LiDAR scan, and hundreds and hundreds of spherical panoramas from every single conceivable position and every conceivable height. This meant we had a full data set to recreate the atrium in full CG.
We knew from some exhaustive previs how much destruction and damage we’d need to do to the building, and roughly where the impact points were, that is, where Wanda was going to smash the floor up, where Captain Marvel was going to get blasted through the wall, where Captain Carter was going to smash against the columns. So we were able to build the geometry accordingly. Rather than building it for destruction, as is the normal process, where everything is closed volumes, that FX can then shatter, we actually built the building brick by brick.
It’s actually architecturally consistent with the real thing. We obtained blueprints and just had a look at how those walls were made. There’s the huge stone blocks and then you’ve got a steel frame and a cylinder block inside and then there’s plasterboard and all the wiring. That was all built into ILM’s model. So, when the model was handed over to the FX team for impact shots, they had to do minimal shattering, allowing them to focus on the dynamic simulation and the additional dust and particulate components.
In terms of the roof structure—which is a bit of an architectural marvel from Foster + Partners at the real British Museum—there was a generative algorithm used in that real architecture to build it. Funny enough, I had given a talk at Foster + Partners years ago and spoken to the people who made the roof. They told me about the math they used. So the procedural method of creating that roof was replicated in Houdini and used to build the model. It’s all completely accurate, in terms of the geometric structure of the roof.
A unique approach to face replacement
There were a lot of stand-ins and stunt doubles for the sequence which then required some quite big and emotive face replacements. Normally when you’re thinking about stunt face replacements, you’re not factoring in lots of facial performance work, because the assumption is that it’s going to be heavily motion blurred, and just a couple of frames. What we found with the Illuminati sequences was that, as the cut came together, we did have some instances where we were looking at a third or half-of-the-frame face replacements and that were on camera for up to 24 or 25 frames, sometimes without motion blur.
This meant that it was going to put our digi-doubles really through their paces, in terms of how much work, facial animation and then lighting we were going to have to do, a challenge we were up for. Our digi-doubles have been built to a very, very high standard, but the exciting challenge was then realizing them as fully emotive CG characters. You tend not to do all that facial development unless you absolutely need to.
What we found was, when we started dialing in quite extreme facial expressions into some of our digi-doubles–the kind of grimaces and snarls that you get during a fight scene–we knew the digi-doubles were going to need a lot of finessing in shots, to make people convincing and photoreal. We therefore looked at what some of the other projects here at ILM, leveraging A.I. and face swap technology.
What I proposed was that we trained a facial capture dataset that we’d shot with the actors as almost like 2D face replacement elements, but we shot that with a camera array from multiple cameras. It was perfectly good as a face swap training set as well. We used that to train the actor’s face onto the digital version of themselves. Rather than trying to do any face-for-face swaps on the footage–swapping the actor onto the stunt double–the way we approached this was that we would start a shot as if it was a digital face replacement, i.e. do match animation, apply that onto the digital double of the actor, render that out in lighting, and then put it into the comp.
Then we would take some auxiliary renders of the digi-double’s face using almost like a head-mounted camera rendered with no motion blur. Once our work was finished, we got back, effectively, a perfect, photographic version of the actor’s face. What we found from that is that we were able to implement so many little touches of realism. As well as improving on the photoreal look, because of its per-frame nature, you get all these little nuances, like little lip curls and eye darts that just add to that organic, random nature of human facial behavior. We then went back through these specular passes from the digital renders and added all of that extra fine pore detail. The end result was great.
The stringiness of Reed Richards (hint: it’s based on Play-doh)
At one point, Wanda turns Reed Richards (John Krasinski) into all these shredded, stringy forms. The starting point there was some reference that was passed to us by Janek Sirrs directly from Kevin Feige. It was someone passing a lump of modeling clay or Play-doh through a garlic press. Janek also had some reference of the Play-doh ‘Barber Shop’ toy, which was a little figure with holes in the head and you’d turn a handle and the Play-doh would come out. That was the kind of look we were going for.
One of the technical challenges was to build a setup that didn’t just look like the body geometry was emitting something else, say, emitting a viscous fluid. Instead we needed to come up with a way to disassemble and shred the real model so that the vertices, the UVs and therefore all the correct texture coordinates, would pass along and be taken with the ‘strings’ of Reed, as they were all being shredded away.
Making it feel like organic material and getting the right amount of flex and balance to the strings, as they come off, was quite a technical challenge. There were cheats, because if you were to actually shred a real human being in such a way, there would be an awful lot of flesh and blood. Of course, this is a Marvel movie and we don’t want to be too gruesome.
The demise of Black Bolt
It’s worth mentioning that both Reed and Black Bolt are wearing fully digital suits throughout. Even when we go into the head close-up of Black Bolt, for him liquidating his own brain, that’s a fully CG outfit he’s wearing.
First “Dr. Strange” rehearsal: landing nailed. Thanks to the incredible stunt team and to producer Richie Palmer for making me feel so welcome so quickly… despite the wedgie. #DoctorStrange #MultiverseOfMadness #BlackBolt #stunts @Marvel pic.twitter.com/ClShgS7TGO
— Anson Mount 🖖 (@ansonmount) May 18, 2022
For Black Bolt, it starts off with Wanda sealing his mouth shut, or rather ”removing” his mouth all together. That was a CG patch that we created using the texture photography from the actor. Then to create the sonic blast that bounces back into his head, and effectively scrambles his brain, before blowing the back of his skull off, the way we approached that was to almost do it for real.
We started off by looking at lots of reference of things getting smashed up in slow-motion. We also looked at human faces in wind tunnels and the amount of ripple it does to your cheeks. We found that it was impossible to sim that and get realistic results at 24 frames a second. So the FX team, instead, stretched out the timeline in Houdini and worked at something like 120 frames a second.
This meant we did the whole thing in slow motion where the face starts wobbling, and then the back of the head blows off. There’s oscillation and contraction to the leather of the cowl, which all happens sub-frame. You don’t actually see it, but I think there’s just something about the way that all that contributes to the motion blur.
I think it was definitely a task worth doing, it really gives some interesting and gruesome results. And hats off to Skywalker Sound for adding that kind of squelch noise, which none of us had heard until we saw it in the cinema. That’s the icing on the cake, just to really up the ‘ooh’ factor.
A new Captain Marvel
We were able to utilize some of the great work that had been done on the first Captain Marvel movie for this Captain Marvel. So whilst our Marvel, Maria Rambeau, is obviously a different person in a different suit, the brief was that her effects and powers should feel consistent with Carol Danvers.
As is always the case, technology and tools evolve, so it wasn’t as simple as just opening up the Houdini setups from Captain Marvel and hitting go. We extensively rebuilt it for our needs. Right down to the basic level. Our model was different, all the geometry nodes were different and then, when you plug it into Houdini, all the emission nodes are broken, so we had to plug everything back in again. And by the time you’ve done that, everything looks different. It was more a case of taking the Houdini scripts, and looking at the processes that we used to generate those effects, and then recreating that, paying close visual reference to the original; but building it to work with our model and our needs.
There are always interesting challenges doing fluid-based simulations on fast moving people and objects, in terms of how much real-world drag and gravity you dial into it. If you’ve got something that’s on fire, but moving 500 miles an hour through the air, you’re not really going to see visible licks of flame, it’s just going to be a streak. So there was a certain amount of cheating of physics to allow her to still have that flickery flame-like binary power, which coats her whole body in the action beat, even when she’s fast moving. There’s probably only 10% or 20% of real-world forces and air drag affecting that simulation.
Stunts and VFX combine for Captain Carter
For Captain Carter’s costume, we had three different jetpacks. There was the hero prop jet pack, which is the chrome, pewter one, nice and shiny, quite heavy, made of metal. And then there was a semi-stunt version of that, which was quite a decent representation, had roughly the same finish, but was made of rigid sponge. The stunt double was okay to wear that for stunt shots, apart from when she was going to roll onto the ground. For big stunt shots, we just had something that was just a piece of sponge. So in most cases, apart from the dialogue shots, where she’s wearing the hero one, the jet pack needed replacement with CG, so we made sure it had tracking markers on it.
There were sometimes physical challenges presented by shooting a stunt where people aren’t wearing a prop and perhaps doing a forward roll. So in some cases we actually had to add a little bit of digital body, as well, to allow the torso to accommodate the jet pack as she performs a roll. And then for clean-up purposes, where she was wearing stunt harnesses or wire rigs, we had a CG version of the suit.
The shield was a prop that was nearly always replaced. That said, there’s a few shots from the lineup, at the start before the fight kicks off, where she was holding a real shield. It has such a lovely anisotropic metallic finish to it. It gives these amazing specular highlights. We were keen to try and keep the real one wherever possible. We used the real shield as a lighting reference throughout the sequence. So when we were shooting the stunts, and for where we needed to add a CG shield, we made sure that we walked the real shield through, changing the angles to all the lights, capturing that anisotropic effect.
Wanda and the robot ‘blood’
When Wanda first arrives at the Illuminati headquarters she takes out the Ultron sentries. We see this in the form of security and camera footage on a big screen, which incidentally was one of the biggest single shots I’ve ever worked on, because it’s basically 17 shots in one. If you look at what’s happening on that screen, there’s multiple POVs of all the robots. Because they’re coming in from all different angles, normally, you do at least something that’s tailored to the camera angle that you’re working with and the FX simulation only needs to work through that particular camera, and animation only needs to work through that camera.
When you’ve got 17 different cameras approaching the subject, from different directions, and everything’s all got to be rendered together, you can only afford to sim once and animate it once and make it work for all those cameras. It had to work from every possible direction. The logistical, organizational and rendering challenges of doing 17 versions of one shot, and then bringing that altogether in comp, were quite interesting.
The robot blood comes from a moment where Wanda decapitates one of the sentries, flips it over her head, and if you look closely in the monitoring shot, you’ll see the squirt of oil or blood that goes over her. There’s only one shot where that actually was required to be digital blood on Elizabeth Olsen. She performed a close-up for that, wearing some tracking markers on her body, and a couple on her face. We used tight match-animation from that to allow us to spray her with digital blood or digital oil, and then that gets revealed in the comp. Other than that, she was very kind to wear that ‘blood’ make-up for the entire shoot. Every morning, her day started with hair getting plastered down, face being spattered with this black blood, and then it all spilling down her body.
What it did mean, though, was that we had to maintain multiple versions of the digital double as well. So we had pristine Wanda, then intermediate Wanda, which is just as she starts the fight, and then the version of Wanda where she walks into the atrium covered in some blood but her t-shirt is still quite clean. Then, as the fight progresses and there’s a bit of tumbling around on the floor, her t-shirt is completely black and filthy. So, that was another version of the digi-double that we had to maintain.
Destroying the atrium
There were a few practical explosions done on set for the close-quarter battle between Wanda and Captain Carter, including when Carter volleys off the statue and launches the shield at Wanda and Wanda then subsequently catches the shield and uses that to deflect Marvel’s powers back towards Carter. Those explosions there were achieved practically on set. They were augmented in CG in post, but the special effects team did a great job of giving us timing reference and look reference. What they did was selectively replace various floor tiles with special breakaway floor tiles that had a hopper underneath them containing explosives and proxy foam debris material, lots of lumps of concrete made out of sponge.
When we were shooting the sequence, we were able to detonate these blasts, sending stuff flying everywhere, which gave us great timing reference. It puts lots of smoke and dust particulates into the air which gives you some quite nice photographic effects as well. It’s definitely worth doing.
The crescendo was a massive challenge for the FX team at ILM. When Wanda and Captain Marvel are locked together into this cyclone of magic versus powers, and they start spinning around, it creates this big vortex of all the debris. I drew out a top-down diagram detailing where they would spin around and where they should end up and what bits of destruction should be where. I’m going to give a hats off here to Koenraad Hofmeester and the effects team, who did an absolutely stellar job. ‘Stellar’ was the working title of the show, funnily enough.
I think Koenraad actually said it was one of the most complex effects challenges he worked on. Not necessarily because of the simulation complexity, but the organizational challenge of managing all these components of the floor getting ripped up, the debris that goes with that, Wanda’s magic rippling over the floor, Wanda’s magic out in the air versus capturing Marvel’s powers, and all the associated interactive passes. It was an awesome feat that ILM pulled off.
A final Easter egg…
If you look inside the atrium, there’s four winged statues, and then there’s another person inside a ring. And then there’s a tall female character, which is, in fact, an Easter egg that I don’t think anyone has noticed yet. The statue I’m talking about is the one that ultimately dispatches Captain Marvel. If you look closely, it’s a female warrior, possibly a female warrior princess. And, of course, if you look into Sam Raimi’s background, he has a famous female warrior princess on his CV. That’s all I’ll say…
