A round-table with Weta Digital on dragons, water and rings.
Just about every Marvel MCU film has a huge third act battle, and Shang-Chi and the Legend of the Ten Rings is no different with its Ta Lo showdown. Here there are ground fights, characters riding on dragons, a wealth of other mythical beasts, and an epic-level ten rings confrontation between Shang-Chi (Simu Liu) and his father Xu Wenwu (Tony Leung).
Behind the visual effects for this third act was Weta Digital, which worked with production visual effects supervisor Chris Townsend and production visual effects producer Damien Carr. befores & afters got the chance to explore the raft of work with members of the Weta Digital team: visual effects supervisor Sean Walker, CG supervisor Joanna Davison, animation supervisor Karl Rapley, and compositing supervisor Sabine Laimer. Here’s how they and the entire Weta Digital VFX crew made the intense battle scenes possible.
b&a: Sean, it’d be fun to get an overall point of view about these enormous battle sequences that Weta Digital had to do VFX for, in terms of the planning for them, what live action they could film, and the first things that you needed to tackle when doing these huge sequences?
Sean Walker (visual effects supervisor, Weta Digital): Chris Townsend, Marvel’s VFX supervisor on the film, approached us really early, maybe six months before they started filming over in Australia. He was extremely collaborative. He wanted to make sure that everything that they were doing on the shoot was all set with us. Usually, I would go on set to help out, but because it was the first half of 2020, things didn’t quite end up panning out that way. A lot of us began this production during lockdown.
There were a few things we had to prepare before filming. We started building the dragon, the Great Protector, quite early on. The reason was that we needed to get the model prepared, so that they could create a buck that the actors would be able to ride on set. Once we created our model, we separated it into fragmented pieces so that they could create different set pieces. They created one for the head and another one for the back, and they used it in multiple different ways. They had a gimbal rig that was able to be rotated so that they could hang from it when it was upright, and they could lie on it when the dragon was on its side.
For the most part, they were really, really good about shooting something for every shot, whether they thought it would work or not. It’s obviously an extraordinarily stunt-heavy show. So they filmed stunties in lollipop rigs and rope rigs, whether we used those plates in the end or not, it just depended on the shots. A lot of times we would have to replace them if they were doing something superhuman, especially if they were hanging onto the dragon and they were suspended hundreds of meters in the air. And because the dragon itself was moving around a lot, we felt like we got a more realistic movement when we were going to digital double.
It was also your traditional Marvel show with lots of edits and changes throughout. We had our own layout edits that we were able to update as the edit changed, to make sure that we were consistent with our continuity, so we would always know exactly how high the two battling beasts were in the sky. We would make sure that compositionally the fight on the Mountain of Souls was always working. Because, if they were to cut shots, all of a sudden the actors would be facing the wrong way. So we used a lot of techvis and layout to make sure that continuity was working throughout. So, that’s pretty much how we of approached sort of piecing everything together before we even really started digging into shots.
b&a: Jo, turning to you, what were the big CG things that had to be solved for these battles? It seems like so many water, creature effects, magical effects were required.
Joanna Davison (CG supervisor): I’d say the biggest one for us was those huge, huge water shots. They were particularly interesting, because the nature of the water was very bespoke. You’ve got these big tendrils of coming over the Dweller in Darkness to hold him down while the dragon is attempting to gain control. And that’s not a natural thing that you see water doing.
After a whole bunch of testing, we got some help from animation to define that movement. And then FX took that and ended up treating the different tendrils almost like individual creatures in their own right. For the simulations there, if you think of the size of the dragon—which was meant to be 110 meters long—if you imagine the size of the water that the dragon’s creating that’s encapsulating the beast, it was just enormous. You have to get that scale across to the audience, but also get those tiny details in as well, so it really truly looks like water. Once we’d created all this amazing stuff, which Claude Schitter, our FX supe, did a fantastic job of heading up, you’ve then got to render it. It’s just an epic, epic sequence to work on, but very technically and artistically challenging.
b&a: I read something about a proprietary caching pipeline for this film. Can you talk about that in terms of rendering out those water sims?
Joanna Davison: That’s part of our in-house renderer, Manuka. The way our renderer works is that it shades and tessellates before light transport. The main problem we hit with those water sims was the sheer memory hit—it couldn’t hold everything in memory at one time. So what we were able to do was go through and cache out the shading and tessellation of the environment and the water and the creatures. You can run that in parallel so you don’t take on that massive memory hit in one go. Then once you’ve done that you just have the overhead to read that in as a pre-process of the light transport; it enabled us to actually render those sequences.
b&a: Karl, I’m going to jump to you. I thought these battles were like an animator’s dream, especially the two big beasts. But also, you have to actually give them certain behaviors and movement, because there’s so much happening. How did you come to that?
Karl Rapley (animation supervisor): Yeah, it was an amazing playground for animation. When I first got introduced to the project, it was like, ‘Here’s two dragons, just two very different types of dragons, just battling it out, while we’ve got our heroes jumping around on rings. And there’s all these little demons going around and phoenixes and foo dogs. And I was like, ‘Oh wow, this is like everything you could dream of as an animator, basically.’ Everything’s going on. You got your quadrupeds, these big wing beasts. Everything! So we had to figure out how, how are we going to deal with this? Because you’ve still got the same time pressures. You’ve still got everything that comes with a Marvel show, an ever-changing edit, but you’ve still got to figure out how everything moves.
It was ‘be prepared’, basically, where you were just going from the start and looking at each creature individually, trying to find a reference for each one, inventing this motion from scratch and slowly building it up. Because obviously you can’t mo-cap a dragon for a foo dog as well as you’d like to. So it’s so much keyframe animation. And, especially for the demon swarms, we’ve got a lot of Massive and Houdini software simulations happening for that, which were really useful for that.
b&a: Were there any references you looked at for the Great Protector, for example?
Karl Rapley: For the Great Protector, we were looking at sea snakes. We were looking at traditional ribbon dancing. It gave some really cool pose movement inspiration. There’s things in film and especially art, where you’ve got really cool serpentine poses. We’re trying to hit those marks, but also something that’s really user-friendly for an animator and fast in the scene as well, so you can actually work with it. We did a lot of testing at the start to try and develop those rigs and settle on something which gave us the best sort of flexibility.
For something like the demon, we were looking at like a lot of bat reference, even like bats crawling along the ground. There’s some really cool stuff like that, because they have to move on the ground a few times, or move across the Great Protector’s face. You’ve got a very talented animation team, and you got to give it to some talented animators, and say, ‘Here’s the idea, here’s the beast. It’s got to break out of this, or you’ve got to destroy this.’ It’s a process, and you collaborate with Marvel. And you find, ‘Okay, this is the right speed. This is the right sort of aggression needed.’ It was a lot of fun though. It’s really cool.
b&a: Sabine, in terms of compositing, there’s just so much going on here. Tell me about the toughest parts of a huge battle sequence like this.
Sabine Laimer: I think the toughest thing was just the sheer amount of different effects that we had to put into each shot. We had everything in every shot. We had magic, we had creatures, we had crowds, we had environments, we had atmospherics, we had particle simulations done in Nuke. And the scripts just got really complex most of the time. So we had to try and template things as much as we could and keep everything modular.
There were some tricky bluescreens as well, especially with the fight sequence stunt work, where we had to replace a lot of body parts with CG because we just couldn’t keep what was in the plate. We had heaps of face replacements. We also had some challenging grading work. With everything shot in Sydney, the plates were really bright and sunlit and we were supposed to make everything moody and dark and stormy. We ended up replacing a lot of the hard sunlight from the plates with CG. We tried a grading approach, where we wanted to keep as much of the plate as we could, but certain things we just couldn’t retain. And so, we had to do a lot of roto work, replace with CG, and repair the edges. Because if you have motion blur over a very bright background, it just eats into the edges. So we had to paint a lot of motion blur back on the edges, that kind of thing.
b&a: Sean, I want to come back to you about making this sequence from that storytelling point of view, in terms of keeping tabs on where everything is up to. Marvel does re-edit and change things up to the last minute, it’s just part of their filmmaking. What has your experience been on this show—there’s so many different pieces of action where you’re in the sky, you’re in the water, you’re on the mountain side. How do you keep tabs on this stuff?
Sean Walker: Yeah, it’s a good question. Marvel are pretty good about continuously sending us new edits. So on this project, more than any other project I’ve been on, every Wednesday, we would get their latest edits. And we would see exactly how the sequence is currently playing out. Karl and I would sit together, we would go through it. We would look at the changes. And if there was anything that was particularly alarming, we would bring it up with Chris Townsend. Those would normally be things where, say, they culled a few shots that would completely rearrange our continuity. Or they would put in completely random shots that fit the story, but were not necessarily the shot that they wanted to use. For example, they’ll take a shot from another part of the sequence, slot it in there, slow it down and say, ‘This is kind of the story we want to tell.
And then Karl will go away, and he’ll create a new version of that shot, or multiple versions of that shot. And we’ll almost postvis a bunch of it. So a lot of the time, we did come down to postvis-ing some of our own work, some of the story beats, just because it was quicker to do that than to go back to say Third Floor and start the whole process from beginning again.
As far as keeping continuity of every individual section, everything’s pretty well sequenced out. So when the dragon and the beast are fighting in the air, we had a layout continuity pass. Layout would make sure that everything was working in three-dimensional space, as well as the camera angles. And then Karl and his team would take those directions and positions and follow them, and then compose their shots according to this overall layout.
b&a: I’d love to ask you each about the actual ten rings fight between Wenwu and Shang-Chi, each from your own perspective, because I feel like there’s such great stuff going on there. Sean, do you want to talk about again from an overall point of view of that in terms of what they shot and what Weta Digital needed to do?
Sean Walker: Yeah, absolutely. So, sometimes they will film something in a particular environment, especially if it’s a stage environment, and then when it comes to actually putting the shots together, they’ll say, ‘Scratch everything we ever built and roto the characters off and create a whole new environment.’ And that’s exactly what we did here.
About halfway through they bluescreened everything out, which meant less roto work for us. They knew they were going to replace it, so they just thought, ‘Let’s help them out and bluescreen it off.’
I think it’s always a bit difficult to try and not get that stagey feel when everything’s fully CG in an environment, while the plate actors were filmed on stage. So that’s one of the challenges, but it did allow us to create some really dramatic shots.
We were able to really define the look of the sequence itself. We went dark and moody and dramatic and kept some of the more prominent features of the Mountain of Souls area bright, so that you’d always get your bearings. The Mountain of Souls gate was red and kind of glow-y. The idea was, you’d always know where you were when you were in amongst this massive, crazy, spin-y action sequence.
It also allowed us to accentuate some of the cooler aspects of the fight. All the rings effects were bright against the darker background. The gate itself was bright against the darker background. We were able to play a nice deep fog and interactive atmos against the darker background as well. It just allowed us to create these really cool dramatic shots.
b&a: Jo, from your point of view, what were the CG challenges of the rings and the effects that come out of them and the different kind of weapons they use out of them?
Joanna Davison: I think it started off with a much smaller number of rings effects in the beginning than we were planning to do. We ended up doing five or six different effects to accentuate the different fight moments. Then we also had to align different colors – the Shang-Chi colors versus the Wenwu colors. We had to develop each of these different looks individually, and that was done largely by FX, working out what different sims not only hit the artistic needs, but also could then be passed on to compositing to hit the final look in shot. So FX ended up rendering a lot of that. Normally, our workflow would be that FX would push into lighting and lighting would deliver to comp. But for these, the renders came directly out of FX.
Then there was a huge amount of work done in comp, using those renders and also using things like Eddy in Nuke. It was just a huge amount of tech and collaboration between FX and comp to achieve the artistic look of not one, but six different types of ring effects.
b&a: Karl, just in terms of the animation of the rings, I wonder, could you do anything you want with them, or was there more of a guide from Marvel for the animation?
Karl Rapley: We did have a postvis guide that Third Floor had done. Before that, when they were planning for shooting, they actually got a Stan Lee action figure and a Captain America action figure and were doing little poses and camera moves on those. And it was actually pretty impressive.
To do the final animation, it was deceptively difficult! From an animation point of view, you’re like, okay, ‘I’ve got some rigid rings. There’s nothing going to be too difficult about that.’ But every hand movement that Shang-Chi does, I mean, the rings had to be behind his movements. He’d put his hands down, and the rings would have to go down afterwards. And everything had to be really choreographed. There was actually quite a lot of iteration, between talking to Marvel and Destin about trying to get that right feeling between the calm Shang-Chi rings and the aggressive Wenwu rings.
They had the stunt performers who would be thrown around on these lollipop rigs, and maybe the arc wasn’t right. So you have to ‘card’ them and throw them higher and just make everything feel a bit bigger. And then they’d be doing these whip actions. We’re like, ‘Cool, that’s a whip now.’ And we’d develop that. And then they go, ‘No, actually we want that to be a buzzsaw now.’
Another example is when he is compressing the rings down, when Shang-Chi’s doing that big atomic blast down. Just getting the look the feel of that was tricky. It was an inner core that was spinning and an outer core that was doing something different. We had to make sure we didn’t screw up the motion blur for Jo and Sabine. We had to get them spinning at a certain speed, otherwise it would have just looked terrible. Lots of little challenges in there that were separately difficult.
b&a: Sabine, in terms of the comp challenges here, I’m interested in the evolution of Eddy at Weta Digital for FX done in comp for the rings fight, but also the other comp elements you had to deal with here?
Sabine Laimer: For the Mountains of Souls environment, we built a lot of canned Eddy sims for little vents and atmospheric smoke, which we could dress in. We had a full layout that we could move through and stack up, which made everything really nicely moody and magical when the light scattered through. We just had to make sure that we were not hazing everything up too much and keep the detail and keep the scale.
We tried Eddy for some interactive ring smoke as well. I think we ended up using Houdini simulations in the end. But it was really nice to explore in that direction as well. As Jo already said, we worked really closely with FX for the rings effects. We had Simone Riginelli, our compositing and effects domain director, linking between the two departments, comp and FX, working really closely with them. We ended up doing a lot of the magic effects stuff in Nuke as well, using Point Render. There were some really interesting outcomes there.
