‘the whole action was shot a blue box with a mirror set in the middle.’
To make the intense Shang-Chi and the Legend of the Ten Rings bamboo scaffolding fight that sees the characters Shang-Chi and Katy pursued off the side of a Macau building, production filmed the actors and stunt performers on a bluescreened set-piece in Sydney, complete with reflective windows and scaffolding sections.
This meant, of course, that the performers would need to be extracted from the plates, and that a massive busy Macau environment would need to be added in to both the backgrounds and as a reflection ‘background’ in the building windows. Handling the VFX work for the sequence was Rodeo FX, led by visual effects supervisor Ara Khanikian. The studio would also deal with weapon replacements and digi-doubles. Khanikian told befores & afters how they made it all happen.
b&a: My first question is, given that the scene was filmed against a building with windows, what were you first discussions about dealing with reflections?
Ara Khanikian: Well, I mean, there’s not a whole lot of ways to do it. I think that the way that [production visual effects supervisor] Chris Townsend and Marvel did it was really the best way. They physically built a three story high section of the building, which was on hinges and could rotate about 45 degrees. It was a mirror set with the bamboo scaffolding set up on it with a couple of planks. Basically the whole action was shot on that—a blue box, effectively, with that mirror set in the middle.
This meant we were able to get all the real reflections of the actors. Obviously, we had to extract them from that mirror set. Just about any time you look at a shot, it’s the main actors, the scaffoldings that they’re holding onto, and then a CG building reflecting the CG city with CG scaffoldings.
b&a: It must have been a mammoth effort to rotoscope and just deal with that set—tell me a little about that side of things?
Ara Khanikian: Yes, the roto was monumental on it, for sure. A lot of it could be extracted; we had the bluescreen, of course. That was all good. But it’s when you start going into it you realize there’s a lot of detail to go through.
When you look at how buildings in Asia are built with bamboo scaffoldings, there’s actual notches between the windows. On set, they had something very close to that but they had to sometimes cover them with plexiglass just so during their stunts and all the fighting nobody would get hurt. That meant between every single mirrored window on set that you’re replacing with a CG one, you have to go in and roto those sections out.
Then there’s the reflections: you have to roto the reflection from the real plates. You’re going to have to add your CG reflection and then you have to dial in the amount of light wrap and glows and flares and this and that.
So, the foreground section was extracted with classic bluescreen extractions with roto and then for all the reflections, we roto’d them in a very standardized way. We knew that a specific layer was, say, just a bamboo scaffolding. Another specific layer would be only the character. Another specific layer would be something else, and so on, so that we could standardize it all a little bit more.
b&a: Was the entire environment that you see either reflected or in the background to shots CG, or did you have any kind of live action?
Ara Khanikian: It was entirely CG. Early on, we were talking about going to Macau physically and shooting plates, getting all the references, the HDRs and photogrammetry and drone shots. What we were thinking about in the beginning was a little more of a classic approach of shooting all of these plates and then enhancing them and projecting them or adding a little bit of CG to them and using it that way. But when COVID started, everything just locked down, and there was no more traveling. So we opted for a full CG Macau.
Production had already done some helicopter surveys as part of scouting, early in pre-production. So we already, knew based on those aerial plates, where the skyscraper would be in Macau. Then we started with Google Maps and OpenStreetMap data to form an accurate representation of the streets. We could do rough photogrammetry captures just to get approximate volumes of buildings. We also even used actual elevation and topographical maps to distort the terrain to represent the ‘real’ terrain of Macau.
Every single building that’s around the main skyscraper is a high rise building built in CG, and those are really accurate to the Macau buildings themselves. For background buildings, we went with a little bit more mid-res models, where we would analyze a lot of Macau pictures and go, ‘Okay, well, the slums in Macau have this particular look. The apartment complex building towers have a certain look.’
We would do generic versions of those and then add enough variation and basically just scatter them into the city. Each has model and texture variation. And since we started with the Google Maps data, we already knew what the volumes and the shapes would need to fill to have that look.
Then we ran some crowd sims and car sims. All the people walking, all the cars–it was this crazy set-up where we had maybe hundreds or thousands of street lamps just lighting up the whole streets. And then, they’re lighting the cars, the cars are lighting the environment, the cars are lighting themselves. They’re lighting the crowd that’s walking around the storefronts. We managed to, with the car sims, to factor in some slow downs and stopping, and then the brake lights would even activate.
What we wanted to do was build the Macau city environment almost as if it was ‘shot’ for real. So that once it got into shot work, you’re not modifying the background. You’re not re-rendering, running sims or relighting specific buildings. We said, ‘Macau looks this way at night, and this is the amount of activity, and this is the amount of animated lights in it.’ This way, we could concentrate the shot work more in terms of doing the set extensions for the scaffoldings, doing the CG building, rendering the reflections and doing all of that and leaving Macau, basically as if it was a background plate.
The scaffoldings were a challenge to match-move as well. Since actual stunt performers were running on them and fighting, they would stretch and warp and move around, and then we’d need to do set extensions for these. Everything needed to be tracked super-precisely.
Interestingly, sometimes on set, there would be two planks, one next to each other for safety. The stunt performers would be running on them. What we would do occasionally is separate them and have a gap between the two to create a little bit more danger. We would analyze where their feet would be in the CG world and then move the planks so that they’re still touching on something.
b&a: There were some really elaborate stunts in this sequence. I’m curious about where you had to do take-overs and how you approached any kind of digi-double work?
Ara Khanikian: The camera work was fantastic for the sequence because it was shot with fairly wide lenses and it was often hand-held. You were always in the action with them. The sections where we did some take-overs were mostly for the wide shots. Some were when it was really just impossible for them to shoot some specific elements tying altogether.
There’s one shot where it’s right after they just step outside, Shang-Chi and Katy, and you have the ten ring soldiers. The ten rings smash one of the windows and it explodes. They’re right there, and then jump out. The guy does a flip and then hits Shang-Chi. We had some shots like that where we’re basically transitioning—we’re starting with the real plate, transitioning to a full CG version and then re-transitioning back into the actual plate.
b&a: What about CG weapons? What the tricky things to do here in terms of weapon replacements and tracking them in?
Ara Khanikian: The tricky thing about weapon replacement, especially on Shang-Chi, is that it looks simple once it’s finished, but there’s a huge amount of work that goes behind it. Every single hook sword that the ten ring soldiers are holding is CG. They had actual swords on set. Some of them were half swords. Some of them were the actual full swords, but we’re adding the blue electric arc on it and it needs to light up and it needs to interact with the environment.
Sometimes we even had some sections where our sims would actually create sparking events and electrical arcs to contact with metals and stuff like that when weapons would come close. So, every single weapon they’re holding is a CG one, and that means every single soldier and their hands and the weapon is match-moved.
We developed a look with our FX supervisor and we were able to easily just spread it into hook sword scenes. But it’s still match-moving every single soldier and what they’re holding. It had to be way more accurate than another movie would allow for, because this needed to work in both the ‘real world’ and in the reflection as well. You can’t just cheat it in screen space, it has to physically be very accurate because otherwise in the reflection it’s going to be off.
In the other sequence we did where Shang-Chi’s fighting with Death Dealer, they had rubber knives or knives that were cut short. Those were replaced as well to give just an extra layer of danger to it.
