The main actors, Gal Gadot, Dwayne Johnson and Ryan Reynolds, were shot separately from the background party guests, then combined together by an in-house compositing team.
When the filming of Rawson Thurber’s Red Notice resumed after being temporarily suspended in 2020 to deal with the pandemic outbreak, one of the sequences that needed to still be shot was a ballroom dancing scene involving dozens of party guests, event staff and a band. It also included, of course, the film’s main stars Gal Gadot, Dwayne Johnson and Ryan Reynolds.
In order to follow a strict COVID-safe shooting approach, it was determined that the principal talent would be filmed first, and then the background guests–many of whom were professional dancers–would be shot separately, and combined with the foreground elements.
This required, however, more than just an ‘A over B’ compositing solution, since the scene had been imagined, like the rest of the film, with an always-moving camera, and included specific dance choreography.
To tackle the problem, production visual effects supervisor Richard Hoover devised a methodology that involved floor tape markers and a metronome click track for the second pass of the background dancers to line up with the main talent pass. Then, an in-house visual effects unit, led by in-house visual effects lead compositor Francesco Panzieri orchestrated the meticulous compositing work to bring the plates together–a blend of rotoscoping, paint work, re-times, hair fixes and more.
Here, Hoover and Panzieri detail the intensive visual effects work involved for the shots for befores & afters.
b&a: Richard, this is a film with explosions and huge visual effects scenes. What made this dance sequence different?
Richard Hoover: Well, I think COVID more than anything. After a hiatus from being shut down because of COVID, we had to find ways to finish the movie. So, that was the number-one priority from that point on. The question was, how do we finish the film as quickly as possible and not jeopardise the look of the picture or the original intent of the picture? And since half the movie had been shot before COVID with a Steadicam/handheld kind of feel to the movie, the initial conversations of how do we do things in two passes drove us to where we ended up for this dance sequence.
We talked about motion control, but most productions are pretty timid about that name and the time it takes and the organization of it and so forth. So, my recommendation to the director Rawson was that we should really try to keep the feel of the picture and allow him to shoot with the Steadicam. In any picture you want your director directing the stars of the picture, and you want the stars of the picture to drive the shots. So we shot the stars first, then the background actors, as two passes. It would’ve been more efficient to do it the other way around, and that was to shoot the backgrounds first and insert the stars, and only in the shots that Rawson liked to make some sort of editing process on the spot.
We went with what really allowed the director to make his picture the way he wanted to, and that was to direct the stars and shoot the shots of the stars first. I came up with this sort of ‘poor man’s’ motion control method to do it after the fact, which meant there was no editing. We were doing full-length takes. And a lot of these takes ended up being over a minute long, which really drove how long it took to do it. If we had the edit, it would’ve been far simpler because the shots in the movie are little snippets of those one-minute shots.
So, to get the backgrounds done, it would often take over an hour, hour and a half sometimes, just to get all the marks to do the poor man’s motion control. And then another half hour or so with the dancers and the extras and the music playback and so forth to get the take accurate enough.
Then, of course, we found out pretty quickly the energy in the Steadicam meant that it was also impossible to reproduce accurately. So I came up with ways of rounding the corners off, if you will, and shooting one lens wider and letting Francesco and his team in 2D compensate for the quick in and outs. We would just try to get a really smooth background plate that was wide enough to encompass the action so that in post, those things could be tracked together.
b&a: So what you effectively did was, shoot main unit foreground plates with the talent, and then you shot the background plates with the same camera operator, or was it a different operator?
Richard Hoover: Well, first unit continued shooting after we shot Gal, Ryan and Dwayne, so we had different operators for the backgrounds. Fortunately, we kept the camera grip through the whole thing. Begrudgingly, he made it through the whole thing, and he was the cornerstone of the consistency of it. In most cases, taking a dolly and pushing it around the floor to the marks. So the operator wasn’t physically trying to move it. We tried that. We tried ‘reproducing the Steadicam’ thing, and a few of them we had to do that way. Like the opening shot in Rome was another Steadicam move where he had to run and move and so on. But as far as the dance floor goes, it was a smoother way to do it with a dolly and that took one part of reproducing the operation away from the operator. He didn’t have to get to a mark and be in a specific place on the floor because he couldn’t look down to see where he was, obviously. That responsibility was on the camera grip.
b&a: When you are trying to line up the moves with the original camera move, are you just using QTAKE or something like that? Are you onion-skinning to try and match the moves?
Richard Hoover: No, I would go through the script notes, and look for the takes where the script supervisor would ‘star’ a take. I also had asked her to make a personal note on there somewhere to say, ‘This was the one I think Rawson liked the best’. Because, for each set up there were eight or 10 takes of it. So we had to pick one as the best chance of success, if you will, because there just wasn’t enough time to do a take for every take of the main unit shot. Of course, the editors were, ‘Why aren’t you doing them all?’ And I was like, ‘Well, we’d be here for six months!’
So I think pretty much what’s in the movie was the star take. There might have been a few cases where they weren’t, ie. the background wasn’t specifically shot for the boardroom.
In any case, I took dailies of the star takes and placed a click track down that was an upbeat on the seconds and a downbeat on half-seconds. So it was like tick, tock, tick, tock, tick, tock, like that. And then gave it to video assist, and video assist played it back. As we progressed and got better at it, I began actually comping numbers onto the playback, so the screen would link the numbers. And he would yell them out, ‘And one, and two,’ so that the camera grip and the operator would hear the numbers.
Then we had the wheat lights on the walls, and as we did more takes, it got better. I actually got production to order some six-by-six numbers. So we taped numbers underneath the wheat lights at key points around the room, which obviously Francesco had to paint out, but on a one-minute take with the Steadicams rotating, panning right, panning left and so on, it was helpful to stay in sync. When we first started there were lights all over the walls. It became very difficult for anyone to remember what the lights meant, even though we’d say, ‘Well, we’ll use green lights when you’re going to the right, or red lights when you’re going to the left,’ like that. We tried all kinds of different methods, but the numbers helped the operator keep track of where he was. This allowed him to know if he was keeping up or getting behind or getting ahead.
We would also record a double exposure of the playback and the live feed in video assist. The problem there being a four to six frame delay between the playback and the recording. So, the recorded DX was off by that amount. So I’d have to make a qualitative assessment of whether the take was ok or not. I would give notes to the operator like you’re always late by a little bit here but ok there and so on. This is how we worked to refine the passes until it was good enough. There was a lot of pressure to complete the work quickly. So we almost never took the 10, 15 minutes it would take for video assist go in and line them up and record a new DX to confirm if the take was exactly right or not. I think the goal was to get about 12 per day, but since the takes were so long, I think we averaged about nine shots, in a 10-hour day.
b&a: Francesco, coming to you. You’re approaching this as an in-house comp team. Where did you start with all this footage and data and details?
Francesco Panzieri: It’s a novel concept that this work gets done in-house by the studio instead of a vendor. It’s quite unusual to have an in-house visual effects unit handling this kind of broad and intricate workload.
Red Notice was filmed at 8K on a RED RANGER MONSTRO with 75-year-old anamorphic lenses that were used on Lawrence of Arabia, which gave this kind of very lush and glamorous look to the scene. I give a lot of credit to Richard for not filming the sequence on greenscreen, which would have certainly helped us creating mattes, but it would have likely also impoverished the final look of the scene in terms of color contamination and reflections on the floor, and broadened our work scope significantly. That choice involved of course a lot of rotoscoping, but once the team became more accustomed to the work, having rotoscoped mattes actually enabled us to target our work on specific hair strands. Whereas instead, if you have a greenscreen footage or bluescreen footage, you don’t have actual single control on the matte information unless you go and just mask individually those kinds of details.
Our in-house team started with three artists. We did about two months of previsualization from January to March on the ballroom scene and the waterfall scene. And then, starting March, we grew up to five artists, and we started hitting all of the production shots in the ballroom sequence.
Because we were all working remotely, hence we had no special pipeline, each artist would download the EXR scans from the color lab, which was Company3. We would then load them up in Nuke with the show LUT and shot CDL’s and start working away. For those shots that needed rotoscoping and painting, we relied on BOT VFX, which was our partner for outsourced roto and paint. We would create a style frame of what we needed to be roto’d and what we needed to get painted. We would then upload the EXRs to BOT, along with the frames where we pointed out what we needed done from them. We’d get the mattes and the paint back, and then we would have everything ready to start the shot.
Each crowd composite involved three steps. The first step was rotoscoping the main actors from the foreground plate. The second step was stabilizing the background plate. The third step was matchmoving the background plate to the foreground plate. There are some shots that are over 300 frames long and the camera is moving almost 180 degrees. That involved a lot of work. Thank God we had a very talented and hardworking team! Most of us had never worked with each other before, however we ended up working together in such a great symphony because we were all versatile in doing our own matchmoving, whether it was SynthEyes for a 3D camera track, or Mocha Pro for 2D camera track and stabilization.
We were also all proficient at painting whenever there was a situation where we had to extend the set here or there. We could manage that by ourselves. Which is what it usually involves being an in-house artist. You are given RAW EXRs straight from the lab, and you need to bring them to final on your own.
One particular challenge was that, sometimes, the background was shot a little faster than the foreground. So you had a motion blurred background, and you couldn’t restore its sharpness of course. Meanwhile, you have a very sharp and focused foreground plate. So, what we did there was matchmoving the background and using perhaps 10% of that tracking data back on top of Gal Gadot and Dwayne Johnson, just to kiss them with a little bit of a camera drift from the background plate that would enable us to contaminate them with a very subtle motion blur that would blend them better with the background. It was a lot of brainstorming and Frankensteining here and there just to make sure that things worked well.
We also had to rebuild all the motion blur on the actors’ edges. When Richard filmed the background plates, all dancers were wearing dark clothes. And in the foreground plate you had just Gal and Dwayne with plenty of bright lights behind them. So we had a lot of halos and bright rim lights, and we had to rebuild the edges affected by those elements because those lights were no longer present in the final shot. BOT was very accommodating in giving us roto in a specific way so that we could just experiment with the mattes, rebuild the edges, and recolor the hair like in Gal Gadot’s case, as she had a lot of hair falling on her shoulder. Her hair would turn blonde because the strands were very, very thin. So we would literally grab the spline that needed the correction, sample the color of her hair from a closeby area on her scalp, and just recolor that spline that way. Sometimes that color also needed a keyframe animation to reflect the lighting changes in a shot.
b&a: Richard, there’s something about the seamlessness of it that comes from, of course, shooting both the foreground and background in the same lighting. As you both have sort of mentioned, you probably could have shot Gal and Dwayne on greenscreen, I guess. Was that actually something that came close to being done at all?
Richard Hoover: Well, since the dancers filled the room, and since there were people sitting at tables against the walls, greenscreen was not an option. At the time we were shooting, the set was needed for the museum scene redressed as museum hallways. We shot the hallways for the dance first so the art department could begin redressing the hallways seen through the doorways of the ballroom. We made it about half way through the ballroom plates before the art department made backings of the hallways and placed them in front of the museum redress when we were shooting the dance plates. I couldn’t take the walls of the set out or move the floor to somewhere else and shoot it on a greenscreen and recreate it, there was no stage space to do that. Also, the amount of contamination coming from a green screen with the camera moves often being 180 degrees or more, you would’ve had green all the way around us.
Although Francesco told you how really detailed and intricate it was to pull off the roto and get it all to work, the idea was to try to keep the line up to where the lights were in the room. I was always trying to get those to line up the same way. So when the light passed behind the actor’s head, the light was on the wall in the background in relatively the same place. It wasn’t always perfect, obviously, and there were times where on a few shots it was so complicated, I wasn’t really sure we had it. So, on weekends, I would get both dailies and do an After Effects comp myself and try to get them to track together, just try to make a determination of whether the plate was usable.
We had a few scenes where we did this approach. I remember a complicated shot in the fireplace hallway–where Ryan Reynolds picks up drink glasses from two different tables. So I had to work out where to have the glasses switch from the background plate to Ryan’s plate and back. Since Ryan was touching the glasses on a surface you have to be dead-on, even if it’s for a few frames. That shot was a very long dolly moving side to side, about a minute long, so extremely troubling. In the film, it’s cut into two pieces, which would’ve been way easier for us to shoot.
b&a: I was also thinking and wondering if you’d had to do anything like this before, Richard, on a previous show?
Richard Hoover: Well, yes. I had used something like this poor man’s process years back when I was a commercials director. I was shooting actors on a greenscreen stage and then building the entire environment in the computer. So, back in the 80’s at Robert Abel, I did a bank commercial where the principal actress was a spokesperson for a bank, and there were people in the background in the bank, and so on. We shot on a greenscreen stage with green blocks representing the teller counter and the desks in the room. Then we hung trees of PVC pipes that we put ping-pong balls on the end of each branch. We hung them from the ceiling, from three wires, so they would be stable. At night we’d have a construction guy come in with a Leica Station and survey every ping-pong ball from three different places on the floor. We did the triangulation from the survey data from the three points to find the nodal point of the camera. Now, it’s 1980-something when this is going on, remember.
So, I had some knowledge of what I thought I could get away with and what it would take to track the camera and what it would take to fit. Even back then, the math to determine convergence is the same as it is now, you’ve got triangles all converging where the nodal point is in the lens, and you want as small a convergence area as you possibly can get to make the camera track accurate. In that case, we got a circle that was about a foot and a half in diameter. Not extremely accurate, really. So then you take all those points and you average it down to a central point and that’s the camera track. So you can imagine it was pretty good, but not super good. The tools we have now obviously are far superior than that, now 40 years later.
Ultimately, on Red Notice, I was relatively confident that we could pull this off and look pretty seamless. The fact there wasn’t any green contamination, that the lighting was the same making the colors the same, made me sure we would be good. As insurance, I asked Rawson not to shoot their feet as much as absolutely possible–I knew it would all help. It didn’t always happen, but when we did see their feet, he mostly use a double and we did face replacements. With the camera’s moving around the actors so much and with the amount of blur you can imagine it would be very difficult to tell if you were really right on or not. Those were all the theories anyway.
Francesco Panzieri: The wide shots where you could see Gal and Dwayne feet ended up being the most difficult ones in the sequence because there was a lot more work involved, because you had to, of course, connect everyone’s feet through the same dance floor. It took a lot more work to do that.
I would also like to mention that opening shot with DJ and the invitation. I actually did the compositing for that one, and I can tell you that the background that Richard filmed for that shot, it was just about perfect. It was 90% there. The shot however is over 500 frames long. So I tried to divide the whole length into imaginary segments, and every 50 frames I would re-time the background plate camera position to match the one in the foreground plate. However, what would happen is that I was now altering the speed of life in that background plate. So waiters and dancers and everybody were now just walking faster and then slowing down, and then walking faster again, and then slowing down. I was like, how am I going to make this work? In the end, it became a blend of things–transforming the plate, just translating it a little bit to where I needed it to be, accelerating the camera movement when there were no people walking by. That shot alone took almost 7 weeks of work.
Richard Hoover: I think what’s super-deceiving is that it looks so simple. But that shot, that opening shot, was the first shot we did with first unit. The original idea coming out of the COVID pause was that we would shoot until just after lunch with the principals, send the principals home, bring in the extras, and shoot the background. We had suggested it was going to take a lot longer than that, but that’s how it goes.
So, on that first shot, I think we were there for 12 or 13 hours, and it wasn’t getting there. It wasn’t even close. The Steadicam operator was sure he could replay his take pretty closely. After a few hours of trying, Rawson had his head in his hands going, ‘What are we going to do? This is terrible. It’s a disaster.’ At that point he’d finally caved in to what needed to be done. I said, ‘You just need to go shoot your actors and leave me the background to figure out.’ And from that day on, we had a second unit.
Now we had weeks more shooting to be scheduled around the extras being on other sets. None of this was in the production plan which meant we were under quite a bit of pressure to get what we needed done quickly, again because the priority was to finish the picture on time.
b&a: I just wanted to also ask about set extensions. Were these necessary?
Francesco Panzieri: It depended on the shot. We didn’t do any 3D work for set extensions. Thankfully, Richard filmed the 2nd unit with one lens wider, so that gave us a little more latitude with all the 2D digital work, which derived entirely from the original photography. Sometimes, in order to blend everything together, we were missing 10% of the frame on the screen right or screen left. So we would go and literally grab that part from a different plate and just comp it back there. Sometimes we had to paint out markers on the floor so we’d clone tracked-patches and comp them in. The markers on the floor were probably one of the most challenging things to tackle because actors and dancers were following those markers that Richard had left, and they were stepping right on them. We had shadow contamination, lights moving left and right.
Also, sometimes, even though things had already been filmed in a certain way, Rawson would want to recompose a shot by repositioning actors, so he would ask us during review, ‘Hey, can we move DJ more screen right? I want to see him more.’ So we would translate DJ’s matte wherever Rawson wanted it to be, and repaint the wall behind him or rebuild Sotto Voce or Gal silhouettes if they had all been shot together and DJ was occluding them. So those were the kind of tweaks that we did for the set extensions.
Richard Hoover: And then, of course, you’ve got spatial problems and scale problems because of where he wanted him to be. Was he in front of or behind dancers that were crossing? Those were pretty tricky.
Francesco Panzieri: Absolutely. During the shot when DJ is walking out after he pick-pockets Sotto Voce’s cell phone, you can actually see he’s dodging an actress walking towards him. But in the foreground plate, he was just dodging no one because there was no actor walking by. So we had to actually roto a lady from the background plate and get her closer to him. A really neat trick that Richard suggested to us was to simulate the shadow of the feather that she was wearing as her mask on DJ’s face. So you actually see light occlusion on DJ’s face happening as she walks by him. And it’s super cool. I love when you are able to manipulate the material in that way.
b&a: Francesco, you mentioned Gal’s hair already. How tough was that to solve?
Richard Hoover: It’s pretty great. We would look at dailies and go…’It is pretty amazing.’
Francesco Panzieri: I’ll tell you this: if I were a visual effects producer on set and Richard had come to me and said, ‘Look, we’re not going to put a greenscreen here. We’re going to rotoscope everything,’ I would have passed out on the floor. We all know that, at the end of the day, rotoscoping is indeed a pricey solution. But it ended up being a better quality solution. If you don’t roto the actors and you just put a greenscreen there, you’re going to have to deal with green spill and green reflections everywhere. So you’re going to have to recolor skin tones. You’re going to have to rebuild edges. You’re going to have to paint out reflections on the floor of the greenscreen and possibly its rig. There’s going to be so much work added to it. And the final quality is very likely going to be lower. Honestly, rotoscoping was the best choice for this sequence.
BOT VFX gave us really good mattes. And it took our team a lot of observing and studying each plate and saying, ‘Okay, look, here Gal had a light behind her beforehand. Now there’s an actor wearing a black dress behind her. So her hair cannot be contaminated in this blondish way anymore because there’s no longer a light to justify that. So we’d have to recolor her hair by making it darker and make it seem like it was just perfect and un-distracting. It definitely took us a while. The team worked on this sequence alone from March 2021 until the end of August. And we had other shots too in the movie. The ballroom sequence alone was over 90 shots, and by the end of the movie, the in-house team delivered over 330 shots, growing up to 8 artists for the final push.
And I’ll add a quick funny thing: when DJ hands over the invitation, production actually gave me a QR code to comp on that. And if you pause the movie and scan the QR code, it will point you to a YouTube video of the movie’s bloopers reel. And I think that by now they have made it public. But for the first two, three weeks of the movie being released, the video was unlisted, so you could only access it through that QR code.
People were freaking out. They were commenting on YouTube, ‘Thumbs-up if the QR code brought you here!’. Originally, it was just like a basic invitation. And then they redesigned it with this QR code and, you know, the whole invitation was motion blurred. So I actually had to fake some sharp focus on the code in order for viewers to be able to scan the easter egg.
