The VFX studio breaks down their face replacement and meticulous compositing process.
In Shawn Levy’s Free Guy, Ryan Reynolds plays Guy, a non-player character (NPC) in Free City, as part of a massively multiplayer online role-playing video game (MMORPG). At one point, Guy and Dude–a muscular, unfinished version of Guy developed for Free City 2–get into a beach-side fight.
Essentially, Reynold’s face and voice would be ‘transplanted’ onto Dude, played on set by bodybuilder Aaron W. Reed. To enable that ‘skinny guy to buff guy’ face replacement to happen, scenes were filmed with both Reynolds and Reed on location, with Reynolds then re-creating the Dude facial performances inside Lola VFX’s specialized lighting rig, which has affectionately become known as ‘The Egg’.
The idea here was that the programmable LED lighting situated on panels surrounding the actor inside the Egg could be set up to replicate the on-set lighting, giving Lola suitable elements with which to do their face replacements of Reynolds onto Dude. Six cameras then captured Reynolds matching the performance of Reed, including dialogue, some of it even new lines.
To find out more, befores & afters asked Lola visual effects supervisor Cliff Welsh, who collaborated with production VFX supervisor Swen Gillberg, how the Egg process worked, and how Lola then tackled the actual compositing.
b&a: Where did you get started with the whole face replacement process for Dude?
Cliff Welsh: Well, they wanted us to do a test first to see what it would look like to have a Ryan’s head on a real buff body double. We actually did the initial test with the body double that ended up being in the movie, Aaron Reed, who ended up being Dude. The skinnier guy we used was Ryan’s stunt double, Daniel Stevens, who has worked on so many of his movies. So, we were like, ‘Oh, this is perfect.’
We did a little test in our parking lot at Lola. We wanted to do one with dialogue, and we wanted to do some action-y type stuff. We did the test, and they loved it. And they loved Aaron, because he was such a buff, big guy. They went ahead with him being in the movie.
They were sold on it, because you don’t really know who the skinnier guy is. However, everyone knows what Ryan Reynolds looks like. So if you’re putting his head or his face on this massive guy, the fear is that it’s going to look odd. But in this instance, I think with the film and this sequence being over-the-top, it helped us. He has this blonde quaffed hair and these big white fake teeth. It’s intended to be an overly-exaggerated version of Ryan, and I think that helped a lot. It didn’t have to completely be this realistic thing. Still, the lighting and everything we did, including what we shot in The Egg, matched the scenes really, really nicely. It helped integrate it all. There was just a little extra freedom because the whole thing’s kind of a gag.
b&a: Before we talk about The Egg, what was the methodology on set for getting the best possible plates for the fight?
Cliff Welsh: Well, I was lucky to be there on set in Boston. I thought in particular that it would be a good idea, especially for the performances, where there’s any kind of dialogue, that we should pay close attention to matching timing. The Adam’s apple moves when you talk. So my idea was, let’s have Ryan do a version as Dude on set, that is, how he would say it to Aaron, so that Aaron would rehearse it before they shot it. And then he would say the line in a timing similar to what Ryan would do. And I think that really helped.
There’s the one shot where Dude says, ‘Catchphrase.’ Well, Ryan actually did that in Boston. He actually did the line there. That became an element, and it helped us for that shot. It meant we already had the exact lighting of where they shot the Aaron plate. And then in general it was nice to have Ryan walk Aaron through how he would do it, just to help out the performance.
b&a: When you are shooting Aaron or Dude plates, what important things are you capturing then? I saw that Aaron had tracking dots on his face, but I was curious about how things like HDRIs or the gray/chrome spheres captured on set inform how you end up programming the lighting in The Egg?
Cliff Welsh: Well, after Editorial approved what they’d shot, we got the plates and then I just matched the lighting manually, which I did with our in-house DP Patrick Flannery. These shots were a little bit trickier for us because this was very specific lighting, being set on a beach. It was extreme, bounced light, and then these really crazy hot spots. Before we did The Egg shoot, we got these really nice spotlights for our rig and we were able to use them for this. It really looked like the sun was beaming on his face.
So, say we have 30 shots or so, it would probably take a full day to actually shoot that with the actor. But to pre-light it, it actually takes about three days. For the pre-lighting, we hire a stand-in that looks like the actor, at least for the most part, with similar features. Height is very important, because when we are pre-lighting, they are actually hitting colored marks which are colored LED lights.
That way, we’re getting all the angles of what the principal actor was doing. Secondly, what we do after we get all the head positions, we then start lighting it to match each shot. So each individual shot is lit on its own. That’s what takes the time. It’s figuring out the right angle, and then figuring out the lighting.
b&a: Tell me a bit more about The Egg setup, itself. There’s a number of LED panels. How are you actually controlling those?
Cliff Welsh: I really rely on Patrick for his vision of lighting it properly. I’m in there mostly to make sure those angles are right and signing-off on the lighting. But getting the lighting dialed in, that’s really Patrick. Maybe I’ll say, ‘Give me a little spot here,’ or ‘Take it down half a spot,’ or ‘Give me a little more fill here.’ We work together, but he’s actually operating the lights through Premiere.
b&a: And, you mentioned getting the angles right and asking the actor to follow those colored LEDs, nut I assume you’re filming with more than just one cinema camera? How does that work?
Cliff Welsh: We have eight cinema cameras that we use. In the center we have three cameras, the middle camera being the B camera, which is my ‘hero cam’. I try to direct everything to this camera. So when I’m setting up these colored marks, it’s all based off my hero cam.
I’d say 85% of the time, unless it’s a very low angle shot, we use the hero cam. But then I’ve got a C camera. That’s right below. Or if it’s a high angle shot, I’ve got a camera right above the B camera, called the A camera. But for the most part, everything’s directed right at those center cameras.
As well, just outside the egg, we have two side cameras called the E and F cameras. And then we have one behind the actor as well. Sometimes, the shot needs to be directly on top of their head, so we’ll throw in a camera, if we can, way up high in the center of the rig.
As far as having those side cameras, if they don’t completely hit the right angle, I might get a little tearing in the projection when I’m projecting the face. And so, having an extra angle from another camera means I can fill in the gap where we don’t have it from the main camera.
b&a: How do you handle eye lines in The Egg?
Cliff Welsh: I’ll tell you a funny story. There was a shot in Free Guy where it’s the big fight scene with Dude and Ryan and Dude has his hands choking Guy. It’s very intense, and he’s straining everything. So when we shot in The Egg, I showed Ryan the plate first. I said, ‘This is a very intense deal. You’re choking yourself, and these are the marks so on, and so forth.’
Ryan’s performance there was so great. I was cracking up the whole time. The way he improvises everything, it’s hilarious. He’s like going for it, not with his hands, but you see his expression. His veins and everything’s going. He’s turning red. He was great. And Shawn Levy, the director, was right next to me. He goes, ‘It’s amazing, Ryan. We got it. It’s good. Perfect. Everything was good.’ And then I had to say, ‘That was great, Ryan, but we’ve got to do it again.’
He looked at me like he was going to kill me. He’s like, ‘We’ve got to do it again?’ I was like, ‘Man, I am so sorry. That performance was amazing, but your face was well too over here. It had to be here, because we wouldn’t be able to capture it.’ And Shawn said, ‘Ryan, are you okay?’ And Ryan said, ‘All right, man. Let’s do it again.’
And he was a trooper. I mean, he could have been a complete jerk. He could have said, ‘Uh, no. I’m done. We got it. I’m over it.’ But then he was like, ‘No worries. Let’s do it.’
b&a: During The Egg shoot, are you doing any kind of slap comps or onion-skinning on the original plates?
Cliff Welsh: Typically, after we do an Egg shoot, we’ll pick a hero shot, and then lookdev it. And then once I have them sign-off on the look, then we can push it through to the other shots.
b&a: Tell me about the comp process and the methodology you’re following to take the plate photography and your Egg photography, and just massage it together. What is the process Lola follows?
Cliff Welsh: After we shoot all the elements, we’ll ingest it into our system. I’ll start going through, we call them selects. I want to make a select from the footage for this particular shot. I’ll scroll through the performance for that particular shot. I see one that I like. I think the angles match. I know the lighting is going to be okay, because we’re pre-lit. It’s more about the angles of performance at this point. We’re happy with lighting.
I’ll do what we call a picture-in-picture. I’ll take the select that I like. I’ll put it in the top of the frame of the plate, and then we’ll send it off to the client and see if they’re okay with that performance. I’ll say, ‘Technically, this works perfect for us. All the angles look really nice. How’s the performance for you?’ Usually, when it’s not dialogue, or it’s not over-the-top stuff, they’ll typically go with what we say.
Now, if it is performance-based, maybe they might say, ‘But we really like this take better.’ And I’ll say, ‘Technically, I don’t think it’s going to work, but we could try…’. I will literally go through all the footage that we shot, and make my selects for every single shot, and then push it over to them, and see if they sign-off on it.
Once they sign-off on the particular select, then it’s tracking the element. We already have the plates tracked. So in this case, it’d be Aaron. We have a 3D scan of Aaron. We track that scan with his face from the original plate. And then we track the element, which we call the select, and then we marry the two tracks together.
b&a: So you’ve had photogrammetry scans of both actors done by this point?
Cliff Welsh: Yes, we’ll either arrange that or we’ll have that from production. And then, in this particular case with Dude, it was really trying to see if the scale worked, because their heads were such different sizes. I like to use the ears, always, of our hero actor, Ryan. I believe if you use the underlying guy’s ears, there’s a distance from your eyebrows to here, that everyone is distinct. No one’s is the same. You might not pick up on it, if I’m using Aaron’s ears. And maybe Aaron’s ears look better because the lighting hits it perfectly or something like that. But something’s going to look even stranger.
Now we’ve got Ryan’s head on Aaron, and then we put the hair on and roto the body back. That had the potential of looking odd. Would Ryan look with like this, if he was that buff? Maybe he would. And here we did have to work with scale a little bit on Ryan’s head. That’s the only thing that really changed. I like to keep pretty much all of what the element is, and not mess with it.
Anyway, they sign-off on selects. This is where I do the slap comp. And we call this a Stage 1. So this is just Ryan’s head on the double, no roto, no color correction. Maybe just one color correct, just get it into the environment.
Then we show them this. Now, 9 times out of 10, it’s ‘Oh, my God. This looks amazing.’ Because it’s just the head. But then you start roto’ing all the other guys’ stuff back and it becomes, ‘Oh, something’s a little bit off.’ Now, I’ve been at Lola since 2010. I’ve been there since the beginning of face replacement. So we’ve learned a lot, collectively. It’s a lot easier just to use the other guy, like a hockey mask. But the more you use of the real actor, the more people will buy off on it.
Another little trick I’ll do is–maybe Aaron’s hairline is a little bit lower than Ryan’s, or maybe it’s higher. But I go where Ryan’s actual hairline is, and then I can warp the Aaron plate. So now I can make that plate get closer to Ryan. And I’ll do that a lot too with head shape. Instead of making Ryan look different, let’s make the double look closer to Ryan’s geometry, and that usually works better.
Below, Dwayne Johnson recently shared his ‘Egg’ experience with Lola VFX during the making of Black Adam.
b&a: And is this all generally happening in Flame, or is it a mix of tools?
Cliff Welsh: I’m compositing all only in Flame. But sometimes we’re tracking with FaceTracker in Nuke. We are using other programs, but the final comps are being done in Flame.
b&a: And you mentioned before, the benefit of having those multiple angles from the different cameras. How often are you using the different angles? You already talked about using the projection and having a slice, which is really beneficial. But I was so curious about how you really use it when you’re in a bind.
Cliff Welsh: If there’s a big dialogue scene, and they missed the mark and I didn’t quite see it when we were shooting, we then need to do stitching. So from a technical standpoint, if I shot something and the angle’s slightly off, it doesn’t always have to be perfect, just in the ballpark. Say it’s a little more off than usual, then I get it to Stage 1, I slap the thing on, and we line up the head, the Ryan head, to where the Aaron’s face head is.
If you just did that out of the gate, but the angle’s wrong, you’re going to have part of Ryan, but you’re going to have stuff that’s just missing. We call it a broken projection, or a tearing. So how do you get that information? It’s kind of like when we shrunk Captain America down. What’s behind him? Nothing, because we have to create it. Well, it’s same principle. If we didn’t catch it with this camera, what’s there? There’s nothing. But we have that other camera. So then we can slice off or roto part of that face and split it in. It’s stitching two cameras together, essentially. It’s saved us so many times.
b&a: I don’t know if fully understand how the seams between faces are so ‘seamlessly’ done. What are the very particular challenges of just selling the seam between hairline or between facial features?
Cliff Welsh: It’s all about using as much as you can from the real actor. Maybe I’m not roto’ing back everything from the double. It’s using as much as you possibly can of our hero actor, and that’s really what sells it. On top of that, we’ve got a pretty good recipe on how to match color. Lighting’s one thing, and it really sells it. We’ve got it to a point where we could dial in the color almost perfectly to the plate. That’s going to create a seamless scene as well, because it’s not going to be a little more contrasty here, and then a little less here, or more so and vice-versa.
b&a: Did you need to render anything? Any hair, or fuzz, or collar?
Cliff Welsh: They did give us the teeth to scan. We didn’t really need them. There were some other face replacements we did in the film. It was some quick actiony stuff of just Guy, Ryan, running, and we did have to create some CG glasses that he was wearing
b&a: Is there anything else, Cliff, that you wanted to mention about that work that I haven’t asked you?
Cliff Welsh: Well, the whole orange-y spray tan on Dude, which is hilarious, was a challenge, because we were now married to that. For all The Egg stuff we shot, Ryan had a little spray tan make-up, but it wasn’t this like over-the-top orange-y tan. So that was a little challenging trying to get that color in there without shooting it.
Become a befores & afters Patreon for bonus VFX content
