How to put two Michael B. Jordans on screen. An excerpt from issue #45 of befores & afters magazine on Sinners.
In Sinners, Michael B. Jordan delivers a magnetic dual performance as the twin brothers Smoke and Stack—two men returning to their hometown in search of redemption. Jordan distinguishes each brother with remarkable precision, crafting unique voices, demeanors and emotional depths. Several sequences feature Smoke and Stack together (and even fighting) in the same frame. For that twinning work, a full gamut of shooting and filming techniques were utilized. These ranged from simple splitscreens to more complicated ones, and to machine learning methods that incorporated a purpose-built rig—the Halo rig—used to record Jordan’s performances for the twins.
“Twinning certainly became a significant part of the visual effects that we undertook,” comments visual effects producer James Alexander. “We made up a decision matrix for how a twinning shot would be done. It was such an analog and tactile proposal, knowing that the visual effects would have to be, for the most part, invisible and in support of everything that was on celluloid, was such an appealing part of the project.”
From Arkapaw’s point of view, the DOP wanted the twinning moments “to feel as if I shot both of these people in-camera, which we effectively did as much as we could. We shot Michael acting in both roles. Michael Ralla did a great presentation on the different ways we could do this. We looked at all of the avenues and then went through the script and honed in on what scenes we would need, say, the Technodolly, what scenes were splitscreen, and what scenes needed a head replacement. But it was always thinking about using as much real footage in-camera as possible.”
Some of the simplest splitscreens involved Smoke and Stack talking to each other or someone else in a locked-off, or a relatively locked-off, shot. “The one shot that comes to my mind,” says Ralla, “is Hogwood (Dave Maldonado) saying, ‘You boys, twins?’, and the reply comes, ‘No, we cousins.’ For that shot, Michael played Smoke standing on the right side, smoking a cigarette. We had a body double, Percy Bell, standing next to him dressed as Stack, also to cast a shadow. Then Michael would go to the trailer, and costume designer Ruth Carter would re-dress him as the other twin, and he’d come back. We’d shoot the other side as quickly as we could so that the lighting changed as little as possible. That was a simple splitscreen.”
When there was dialogue, these splitscreens then required sound synchronization and the use of video playback to broadcast the lines Jordan had said previously, with the lines of the double scrubbed. An on-set team made that possible, also adding in ADR beeps where there were more than just the twins in the scene so that Jordan knew when to start his lines.
Occasionally, Jordan acted against a tennis ball—to some hilarity on set—while at one point a monitor on a stand was employed for Jordan to act against so that he could see his previous performance. “The problem was,” reveals Ralla, “Michael is also a director, so when he saw himself delivering his previous performance, he wasn’t acting with it, he was actually evaluating his performance using his director brain. So that monitor went away. It wasn’t working. Then we played back his voice, but it was really weird for him to hear his own voice. So then we had the double saying the lines of what of the A-side, so that it would feel more natural for Michael and that’s how we moved forward for those easiest cases.”
For the next step up in terms of kinds of twinning shots—with moving cameras—the Technodolly approach was undertaken. It allowed for recording moves and the recalling of moves in sync with audio and video. “It’s such a mobile piece of kit as opposed to a more traditional kind of motion control on rails,” notes Alexander.
The standout Technodolly twinning shot was the moving shot of Smoke and Stack as they wait at their car for Hogwood, and share a cigarette between them. Stack rolls the cigarette, and then hands it off to Smoke, who grabs it, takes a puff, hands it back to Stack, who now takes a puff, and then hands it back again. “That shot is a little bit of a unicorn among the twinning work,” says Ralla. “It’s the first time we see them and it’s also a reveal. First we see one of them, and then we see both, so it had to have camera movement. It’s also a great way to see how Michael was developing different body language and mannerisms between Smoke and Stack.”
“Even though it had camera movement and even though it had interaction,” notes Ralla, “I didn’t want the first reveal of the twins to be a body double with Michael’s head or face. I wanted to use a Jurassic Park approach. On that film, there’s only something like 63 digital dinosaur shots. But every single appearance matters, there’s a lot of other dino moments, but they’re implicit, like the ripple in the glass. So, we wanted to do something very similar, where we established tentpole shots at only certain moments in the script, that is, where we’d just go the hard route and actually shoot Michael full-body, twice, even with interactions and lots of camera movement, just to put in these anchors of real-Michael as much as possible.”
The cigarette swap was a shot that first went through a series of testing six weeks before the start of principal photography. “For a variety of different reasons,” states Ralla, “I wanted the crew and Autumn to get used to the Technodolly and the whole workflow. I’m also a big fan of doing a ‘Pepsi challenge’, where we try different options and compare them in the same exact setup, or we shoot a real reference and then build a CG equivalent and compare the two.”
“The idea behind the Pepsi challenge for the cigarette swap,” says Alexander, “was that we could put them next to each other when they were both finished and hopefully Ryan wouldn’t be able to tell the difference. Importantly, I learned things about the Technodolly that I hadn’t appreciated before that, for instance, you can uncouple all of the motors and you can move the head around like a Steadicam, almost, and that move is recordable and repeatable. It was an incredible tool.”
The cigarette handover test was filmed as a swap happening multiple times, using the Technodolly. “It had a crazy number of hand-offs,” recalls Ron Tatham, credited as Technocrane operator. “I think there were at least 10 hand-offs. It was really rad, but we were just trying to figure out how far we could take this. That camera test was also about getting all departments in a rhythm and figuring out the workflow for how it was going to be shot for real.”
At the same time, the team shot everything that would be needed to do a digital face replacement. The splitscreen composite was given to Storm Studios (which also carried out the actual final shot). For the digital face replacement side of the test, Ralla and Alexander had brought on Rising Sun Pictures.
Earlier, a range of solutions for digital face replacement had been explored, including volumetric capture and other capture techniques. One issue, however, that immediately became apparent for Ralla was the desire of the Sinners filmmakers to “focus on story and be in the moment. There was no chance to have Michael do a take and then say, ‘Okay, and now we need you to go in the trailer and then re-enact the same thing you just did in isolation, or to do it in a completely different location some other time. This was just never going to work and it would not give us the best performance.”
Ralla came to Rising Sun Pictures knowing they had developed a toolset called REVIZE that uses machine learning techniques for face replacement. “Rising Sun visual effects supervisor Guido Wolter had sent through a spec sheet of what they needed in terms of capturing the actor. The most important thing was being able to do it there, in the moment. And that became the Halo rig.”
The Halo rig was a 10-camera rig on a ring that was shoulder-worn. “We made it like that rather than being head-worn because we wanted Michael to still be able to fully articulate and move his head around,” says Ralla. “But we also wanted to shoot him from as many angles as possible. It was to give Rising Sun an abundance of training material to use as a dataset for the REVIZE workflow.”
A company called Wild Rabbit Aerial built the Halo rig. Interestingly, they are known for drone work for films, but they also had experience with camera arrays, which is why Ralla went with them. “They built this carbon fiber ring and the harness, which was very light. It needed to be nimble, and we needed to be able to do this really fast, because that was a key aspect to making the performance successful. The Halo rig was built prior to the Pepsi challenge, and used during that test shoot.”
The cameras in the Halo rig were Insta360 Ace Pros, which featured 1/1.3-inch sensors paired with Leica‐designed 16 mm f/2.6 lenses. “They shoot 8K with a modified firmware that also came from Wild Rabbit that allowed us to actually start them in sync, even though they were not genlocked,” advises Ralla. “We also found that they could alter the firmware to use a log color space.”
The close proximity of each department head on Sinners meant that Ralla could quickly ask his fellow filmmakers about how the Halo rig might impact their own needs. “Michael would come in,” shares costume designer Ruth Carter, “and say, ‘We have this idea for a Halo rig. Do you have a replica of the costume? Do you have a mannequin we can borrow?’ They’d be part of the costume fitting to understand the costume a lot better and to test the rig. We could really rally behind each other’s ideas, we could support each other in so many small ways.”
The Halo rig data was accompanied by a process Ralla referred to as the ‘Foxtrot’ where Jordan was placed next to a body double and the pair would go through a series of head positions and expressions. “It was like a simplified FACS set, turning their heads and saying the same lines. That was shot on film and, again, on location, on set, in the accurate lighting, right after the take. We would finish the A-side, then we’d do a Foxtrot and a Halo. Then we’d finish the B-side, and do a Foxtrot and a Halo for both the double as well as Michael.”
In its reference gathering exercise, Rising Sun Pictures also received traditional photogrammetry-style scans of Jordan used to build a digital double. Finally, a controlled shoot off-set was undertaken of Jordan using standard full-frame digital cameras. The idea here was to acquire high-fidelity data of the actor and his doubles beyond the captured dataset.
Rising Sun’s REVIZE workflow was then utilized. “Leveraging our machine learning capabilities,” states Wolter, “the team employed novel-view-synthesis—-a gaussian-avatar style approach—to create and reproduce angles of our hero as needed. Similar to the diverse range of techniques used during principal photography, we also employed a combination of machine learning, CG and advanced compositing methods to convincingly portray Smoke and Stack together on screen.”
The resulting twinning shots from the test—a splitscreen and a machine learning-based digital face replacement—were both deemed successful, while also helping the VFX team determine how these kinds of shots should be done for the whole film, depending on the action. “It confirmed exactly what we were saying,” points out Ralla, “that the body language of Michael mattered so much that you could tell when we just put Michael’s head on the double, in that it wasn’t in line with all the other performances. So, anytime we saw more than just shoulders, or anytime we saw Michael, especially standing still and not interacting, we would shoot him twice, no matter how hard it was. Just because we knew the body double hadn’t gone through the same development and training process, developing Smoke and Stack’s body language, and it was key to making all this successful.”
“The other thing that we learned from this Pepsi challenge,” continues Ralla, “was that a face replacement, in most cases, wasn’t enough. Because Michael is a director himself—and he’s directed himself too—he knows what he looks like from behind, and he knows what his ears look like, which not a lot of normal people do, I mean, I certainly wouldn’t. So the first time he saw a face replacement, he was like, ‘Those are not my ears, that’s not my hairline.’ He could see it was the body double. So, sometimes Rising Sun had to do a full head replacement, including the hairlines and sometimes even full hair.”
There’s more on Rising Sun Pictures’ twinning shots, below. But first, a look at how the final cigarette swap shot appearing in the film was achieved. It ultimately became a 1350 frame visual effects shot with a wealth of interaction, as Smoke and Stack hand the cigarette back and forth to each other.
The Technodolly was employed for the various passes involved, essentially an A-side with Jordan as one of the brothers, and then B-side as the other brother. A carbon fiber rod positioned on the car the pair lean up against was used as a spatial marker for where the hand-off could occur each time.
What the Technodolly enabled was precise repeatable moves, as Tatham explains. “We would hit record for the A-side, and whatever was filmed—whether it was a pan, tilt, zoom telescope, push in on the track, boom—the Technodolly remembers everything. Then we put it back in playback mode and run through the B-side.”
“Here, we had audio beats on the B-side,” adds Tatham. “We had a QTAKE system as well. That gave us a 50-50 gradient with the hero take. In real-time, we could see Michael A and Michael B as a live comp and see if they were interfering with each other and where they lined up. You could see, oh, he was a little fast, he was a little slow, he leaned a little bit to the left. On the B-side, too, not only did we have audio in sync with us as we were playing through the move, but we also had a way of triggering the QTAKE to have a live overlay.”
The QTAKE video playback system was overseen by video assist operator Dan Furst, who implemented a GPIO (General-Purpose Input/Output) pin trigger to aid in coordinating the Technodolly moves, video playback and sound playback. “It’s an essential piece of the puzzle when it comes to making everything go,” outlines Furst. “It connects to QTAKE through a USB. I would put a point for a GPIO trigger in the timeline, and when you play past that point, you can tell it which triggers to activate. It ran to the Technodolly and to the sound department and it would trigger their individual boxes. When you’re shooting film, especially large format, timing was important. You really can’t waste any time on the film.”
“For the A-side take on the cigarette swap,” continues Furst, “we would roll sound and then we would have the slate ready with the bloop slate that would trigger everything, then they would hit the clapper sticks, have the bloop slate right away, and then playback would start. When the bloop slate would go the first time, it’s just a recording for me. But when we would do it the second time for the B-side half of the twins, I’d have then queued up the bloop slate in QTAKE point for the GPIO trigger and the sound department.”
Meanwhile, audio beats implemented for Jordan were orchestrated by production sound mixer Chris Welcker. “We had been thinking about how to help get the timing right when they were shooting these shots,” shares Welcker. “We figured, Michael B. Jordan was familiar with ADR and the series of beeps used to cue them as to when to start delivering their dialogue. So we thought, maybe we could create a series of beeps that could alert him to when the timing of the hand-off should happen and give him the ability to preempt that timing by giving three beeps before the fourth one, which is when the hand-off has to happen.”
Using a Pro Tools set-up on location, Welcker looked to integrate these beeps with Furst’s video playback system. “Dan was essentially recording a bloop slate. He would physically press a button and it would trigger a light for the start of the camera move. His system then sent out that GPIO pulse which meant we could sync up to his system. We had to figure out a way to take that pulse and convert it into something that could be interpreted by the Pro Tools system as essentially starting our playback system.”
In addition, Welcker was able to take the recording of what Jordan and the double had said on the first A-side, and play those back on the B-side. “That opened up a whole can of worms, because now we were essentially having to not only record the dialogue, but we would also have to edit that first performance where we were using the Technodolly and we would have to essentially cut out anytime his stand-in Percy speaks, to make space for Michael to do his thing.”
“It took about 25 to 30 minutes for Michael to leave and to come back as the other twin,” adds Welcker. “We had to take the mic off him. Then he would go to hair and make-up, then wardrobe, and come back as the new twin. We’d have to put a new mic on him. When he would come to get his mic, he’d be there standing next to our system and we would say, ‘OK, this is what we came up with, and we could maybe audition with him some of that playback just so he could kind of hear himself and get prepared.”
The cigarette swap shot took 18 takes to master. The video tap and telecine’d transfers would go to editor Michael Shawver. “I lined it up and I watched it,” Shawver says. “Michael is such an incredible actor and he brought it on both sides. I’d never seen him so locked in. Each take would have a little bit of different nuance to it. Part of what I do is, when I take it in and I do a quick comp of those shots, and then watch it back, I put myself in the mode of an audience member. And sometimes, it’s just how it all feels. In this instance, we ended up switching out one of the takes to the one they shot to. As long as that point of connection with the cigarette was lined up—basically the move had to stay the move because the camera obviously couldn’t look weird and separating—but once we got into that shot, I could then manipulate one side or the other as necessary.”
The other tricky part about that opening cigarette shot, from Shawver’s perspective, was establishing some momentum at that point in the film. “You really want to balance getting the story going, while also showing this spectacle with the twins. But we don’t want to overstay our welcome. You want to have it so the audience says, ‘Oh, OK, I can see that. If the audience starts to think for themselves, not when you want them to think for themselves in later parts of the movie, then that can be a bad thing and it can start to unravel. You don’t want the audience to be thinking about the effect. So, we found in the final cut a way to cut back and forth to show an empty road, to communicate that they’re waiting for somebody.”
Storm Studios then took on compositing the final cigarette swap shot. The studio had to break down each handover of the cigarette, each exhale of smoke, including where the takes might not quite line up and where the lighting changed between takes. These would be solved with re-times, warping, reconstructing the plate, and morphing.
Four compositors from Storm worked on the shot, splitting between the three cigarette handovers, and then the clean plate and paint-out work. “We set aside three or four weeks of comp time to do this,” mentions Storm Studios visual effects supervisor Espen Nordahl. “We started with some mock-ups firstly just using the telecine QuickTimes. Then once we got the scanned plates, we had multiple takes, so we would start to digging in to see how we could use them. They were all similar enough that we could use an arm from one take, the legs from another. We had to reconstruct the cigarette itself, because the angle was different in each handover.”
In terms of the cigarette smoke, there was exhale each time, ie. smoke in the plates. “That was a challenge because sometimes the wind was blowing in the opposite direction from the other take of Michael,” observes Nordahl. “In the end we did remove almost all the practical cigarette smoke, and then did a smoke sim in Houdini. There was a clean plate for the background, which worked great for any smoke that wasn’t on top of Smoke or Stack. Then everything on the face, sometimes we’d just paint out on key frames and morph between them, then put our smoke on top.”
Twinning visual effects was also necessary for some driving sequences featuring both Smoke and Stack travelling through cotton field roads. In one scene, the twins are shown sitting in the front seats of the car, with Sammie (Miles Caton) in the back. “It got challenging because Smoke is driving,” notes Ralla. “In fact, whenever they’re together, he’s always driving. But how do you do that when we’re replacing them? So that’s where we shot with a tennis ball on a process trailer.”
The overall process consisted of first shooting Jordan as Smoke, who would be driving for real in a real car, with a hard-mounted camera. When the camera was then placed on the other side—for the Stack side of the shot—the action moved to a process trailer, where the car would be loaded onto the back of a truck, slightly elevated (the horizon line would be fixed later, alongside the generation of the correct backgrounds). The tennis ball came in to minimize occlusions.
“What we settled on was recording the A-side with Michael as Smoke, and Percy there as the double for Stack,” explains Ralla. “Then the sound department—who were following along in a moving van while doing all of this—scrubbed the double’s answers so that Michael could speak them on the B-side. Then the double would re-create Michael’s, or Smoke’s, lines for the shooting of the B-side.”
The hard-mounted camera brought with it its own challenges in terms of compositing the two takes together, owing to vibrations and small differences between the plates. “We actually had Storm Studios explore techniques to remove motion blur with CopyCat in Nuke, which was doable to some degree,” advises Ralla. “But what was really interesting was that the vibrations would sell the authenticity of those shots. Where we landed was that we combined the vibration from both sides and that would become the plate vibration. We analyzed the vibration from both plates, sometimes the spikes were exactly the same, at the same time. If they weren’t, then we would add the vibration from the other side.”
For other twinning shots where Jordan might be finding that his performance sometimes wouldn’t fit in the time that was allotted based on Percy’s performance, Welcker’s team had to pivot quickly. “We started exploring pre-recording Michael’s dialogue for the twin that he would eventually play in the second setup. My music playback operator Ryan Ferris would literally trigger pads on a sampler, a drum machine essentially, and we would have the lines of dialogue from Michael pre-recorded and broken down onto these different buttons. And so, he would essentially perform against himself while Percy there, but not actually speaking. To add even more complication to that, there were those times where they wanted to do all these things while we were also in a car driving, doing this wirelessly with earwigs in the actor’s ears! We sometimes had a dialogue editor in a van putting together these performances and playing it back through my system all while driving wirelessly through separate vehicles. It got pretty complicated, but by the time we were done, it was like, man, what can’t we do?”
At the climax of the film, Smoke fights Stack, who has been turned into a vampire. This, of course, required the action to appear as if Michael B. Jordan was fighting against himself. Stunt coordinator Andy Gill was called upon to choreograph a unique fight between the brothers.
“We’re going back to the 1930s, and so right off the bat I started talking to Ryan about how both the brothers went to World War I together,” recounts Gill. “I looked up how people trained in World War I, and it was basically grappling, wrestling and boxing. It was not going to be a martial arts-type fight that you might see today. Tim Bell was the local stunt coordinator who came in and helped with a really great team. One guy was a wrestler, one guy was a boxer. What we like to do is put together what we call a select list where we’ll videotape 10 or 15 different moves and show them to Ryan. We finally cut it down to put in the moves that he likes into the fight, and then we also would do a stuntvis version. I’m used to big wire pulls and big fight scenes, but I had to tone it down and make it a realistic period piece.”
Gill then worked with Jordan to train and learn the fight. The actor was, unsurprisingly, busy with shooting scenes, but anytime he was available on the stages, he would visit Gill in a warm-up area so they could teach him the fight. “He learned so quick,” marvels Gill. “The only reason it really worked was because of the dedication he had to it and the input he had. A lot of times we would be doing something and he goes, ‘Let me think about this.’ Sometimes we would have one brother just trying to kill the other brother. Michael would say, ‘I’ve been having this talk with Ryan. He goes, we’re brothers. Even though he’s a vampire, he is still a brother. I don’t think he would just outright kill me if he had the chance to, right off the bat. He’s going to think twice about it.’ So we put a few of those pauses in there.”
Jordan performed the fight against his stunt double Devante Antonio Thomas. “Devante mimicked Michael B.’s movements as much as he could,” relates Gill. “Michael B. would have to fight the double and then we’d switch, and then he would be fighting a double again, but as the other twin. That meant he had to learn two different fights, two sides of the fight, which is huge. He knew the moves from the first half, so he knew how to react to them.”
Gill was particularly impressed with how Jordan adjusted his performance depending on whether he was doing the Smoke side or the Stack side. “The way he changed over, not just via wardrobe, but his attitude, I mean, if he had been naked you could tell which one was which! Just the way he held himself, just the way one hunched over a little more. One was rigid and straight up, one had a little swagger. Every time he walked in, I know exactly who it was. It was phenomenal.”
In addition to these changeovers (which occurred with the stunt double as well), Jordan would go through a round of shooting with the Halo rig. “Michael B. would do a fight and then once he does the fight, before he went to the other side, they put the Halor rig on him and he did all these movements,” explains Gill. “Then they put that same thing on the double and he would go through the movements.”
Rising Sun Pictures then handled the twinning work for the Smoke vs. Stack fight. By its very nature, it included a range of close-combat where the original plate would be Michael B. and Devante, and then need to be finished with two lots of Michael B. Jordan in the frame in close-combat.
For these fighting moments, the general approach was for Rising Sun to replace Devante’s head with Jordan’s using their REVIZE workflow. However, Rising Sun also completed some body alterations, simply because Devante had larger biceps and deltoids. Having purpose-captured models of Jordan as both twins, and of the stunt double, meant that the visual effects studio could complete these kinds of augmentations. “They would always start with a rotomated version of the digi-double,” outlines Ralla. “Then they would run their ML tech to turn the digi-face into Michael’s face, but always using the performance from the Halo rig from Michael’s performance, not the stuntie’s performance.
The shots were incredibly complex, notes Rising Sun’s Wolter. “The art of digital twinning is inherently binary. It either works seamlessly or it doesn’t. That makes it both incredibly unforgiving and immensely rewarding when it works. Having Smoke and Stack physically engage—punching, grappling, grimacing, sweating and bleeding—introduced a dynamic layer of complexity. Each shot presented a unique challenge; there was no ‘one-size-fits-all’ solution.
157 pages of VFX and filmmaking behind the scenes on ‘Sinners’
Over the course of the film, Rising Sun Pictures would rely on its REVIZE workflow for a range of different shots, some as complex as the fights, some with minimal interaction, and even some where the usual Technodolly approach to filming had not been possible. The studio ultimately processed around 125,000 unique faces, a data set that totalled more than 52 terabytes of reference material. Importantly, while machine learning was at the heart of Rising Sun’s work for their twinning shots, multiple techniques were used, and always with the goal of preserving Jordan’s original performance.
“We approached the shots with the goal of seamlessly switching between techniques,” states Wolter, “whether in-camera, CG, machine learning or compositing, so the audience could stay immersed in the moment without detecting the illusion. This work demanded everything from our toolkit and we believe the end result is a compelling blend of skill and technology. We’d like to believe you’d be hard pressed to find where our artistry begins and the illusion ends.”
Get the full mag:
PRINT: https://www.amazon.com/dp/B0G5JJLHRS
DIGITAL: https://www.patreon.com/posts/issue-45-sinners-145307590
SUBSCRIBE: https://www.patreon.com/c/beforesandafters/membership
