Behind ILM FaceSwap and the process the VFX studio followed to turn Ford from 80 to 42.
Over the past few years, Industrial Light & Magic has been called upon to deliver visual effects for several de-aged characters. Think: the mobsters in The Irishman, or Luke Skywalker in The Mandalorian and The Book of Boba Fett.
With each de-ageing VFX challenge, ILM has been refining its techniques to pull off the shots, from the photography and capture stages, to utilizing 3D digital doubles and machine learning techniques.
The accumulation of that history of de-ageing work is most recently apparent in James Mangold’s Indiana Jones and the Dial of Destiny, where ILM was responsible for bringing an 80 year-old Harrison Ford back to his early 40s.
To do that, the team at the visual effects studio relied on its suite of tools and workflows known as ILM FaceSwap, overseen by production visual effects supervisor Andrew Whitehurst and ILM visual effects supervisor Robert Weaver.
“ILM FaceSwap can constitute anything from traditionally shooting a separate plate and then tracking and putting that in, down to fully replacing the face with a CG asset,” describes Weaver, who shared with befores & afters the process ILM followed to take Harrison Ford’s live-action on-set Indy performance and turn it into the younger Indiana Jones.
It starts with performance
Weaver credits Ford with delivering an extremely convincing 40-year-old version of himself on set. “We were incredibly fortunate to have Harrison driving the performances here because he is just fantastic at any age in delivering performance. When he was directed to give a more youthful type performance, he was on cue immediately. That included body posture, it included attitude, and all of these things conveyed into the work that we would do, specifically the facial performance.”
“We were handed a very fortunate set of cards in that regard with having Harrison deliver these performances,” reaffirms Weaver. “Because we knew that everybody was happy with the performance, we had to just do the de-age work and have the same conveyance of what he was delivering, but 40 years younger.
During filming, Ford performed with tracking dots on his face. This helped ILM to track and register the way his face was moving, a process aided by the visual effects studio’s proprietary Flux system originally developed on The Irishman.
“Flux is comprised of a couple of infrared cameras that ride on either side of the main camera,” explains Weaver. “Then we also have witness cameras. All of these things are terribly important to be able to reconstruct in the computer what happened on the day on set. Once we have that, we go through the process of doing the rigid tracking.”
Crafting the asset
The next step was building a CG asset of Ford’s head. This began with a separate Medusa scan of the actor. Medusa is ILM and DisneyResearch|Studios’ proprietary facial capture system made up of a mobile rig of cameras and lights and software that can reconstruct an actor’s face in full motion.
“It involved going through all the Facial Action Coding System (FACS) for facial expressions to hit all the main expressions,” says Weaver. “We built a library with those main expressions and then that material is used as a base to build off of by our ILM artists.”
This scan gave ILM a present-day Ford, which the studio then had to turn into a younger version of the actor. A scan actually existed already at ILM from production on Indiana Jones and the Kingdom of the Crystal Skull. However, it was not quite the right target age.
Notes Weaver: “We leveraged parts and pieces of that scan and then adopted it as necessary to give him the target age of 42. Then, all of those FACS shapes, all of the expressions, had to be applied to the younger version of him as well. In essence, we had two versions. We had the present day and then the younger version. The shot would be solved first with the older version so that we had a good ground truth.
Then we would put in the younger version and see how it felt, how the performance was conveyed, and then started working it up.”
As development ramped up, ILM also worked on a proof of concept for the younger Indy. “We took some older shots from movies and replicated those,” recalls Weaver. “It was our ‘Pepsi Challenge’. Then we’d also be playing with the machine learning aspect of it, and we went through various rounds of how that works and what we needed to do to make that as successful as it could be.”
Incorporating machine learning
The machine learning side of the process–which is part of the ILM FaceSwap suite of tools–started with a search through past age-relevant Harrison Ford performances. “We would find frames from past Indiana Jones movies and match the orientation of the current shot,” discusses Weaver. “It gives us a good target reference to understand how Harrison looked back in the 80s, frame by frame, with various expressions.”
One area that Weaver says machine learning techniques helped with, in particular, was giving ILM “a leg up on getting the facial performance much earlier than we otherwise would be.”
“We took all of the initial circle takes from the performances and we did quick low-res renders of those and delivered them with a bounding box around them,” continues Weaver. “What it gave editorial was an age-appropriate target for the edit so that they could make much more informed decisions. We were able to turn those around in two to three days. During the course of processing the circle takes it also informed us as to how the performance was translating to the younger age as well.”
Weaver says the inclusion of the machine learning steps stemming from the trained model of historical Ford imagery also proved helpful in reaching the right performance. “It alleviates some of the more painful steps of having to iterate to capture the nuances of the performance, per se. It by no means was just, ‘Okay, we’ve got a face that is being captured and we’ve trained a set of imagery to help in that regard.’ We had to go in and do various blends between different performances and different techniques that are employed within a single shot. There were times that we would have a part of the frame that we would salvage and be able to incorporate with a little bit of paintwork, for example. Every shot was its own unique kind of idiosyncratic type of work. There is no recipe.”
Indeed, that is a challenge often faced by visual effects studios in crafting de-aged performances or even crafting any photoreal digital double when relying on reference. The fact that a character or digital human can look different in different angles, in different lighting, and even between shots means that sometimes the reference does not always give you what you need for a specific shot.
“Sometimes you don’t get a nice clean profile from some of the techniques employed,” comments Weaver, “and you’ve got to rely on other ways to achieve that. It could be shape work, that is, taking your asset and just contouring the profile to be the correct shape. For example, we relied on the CG asset for the beard stubble, all the profile edges, the hair, the neck, the ears and even eyes at times. There were occasions where we were able to use the eyes from the plate, which was tremendous, because, again, that’s Harrison.”
The fine details
The de-ageing visual effects were concentrated on Ford’s face, but ILM also had to carry out ‘youngification’ on certain body parts. “We were incredibly fortunate, again, with Harrison being just in phenomenal shape,” says Weaver. “He is so fit. There were maybe two or three shots that we ended up doing a little augmentation on him to make him stand up, or throw the shoulders back a little. Otherwise, he was on cue and just gave that youthful representation remarkably.”
“We would de-age the hands,” adds Weaver. “As you get older, age can show, or the veins protrude more. Those types of things have to be dialed back. Otherwise, it was just dealing with what we had, just making it appear younger.”
The other intricate side of the work of course included attention to detail in lighting and compositing. One of the first shots filmed and turned over to ILM was the initial reveal of young Indy during an interrogation scene when a bag is removed from his head. This hero shot would be one that the VFX studio honed its de-ageing process on, partly because it also had the added challenge of the face being lit by torch light.
“That was a challenge for the lighting artists to come up with the same look as exactly what was shot on set,” notes Weaver. “It took a few rounds to get the lensing artifacts right in terms of how the torch light falls off, and the pattern in it and everything else. Then, as you can imagine, in the CG skin, the way that light interacts with it is dependent on all of the various lighting conditions with light and shadow. The more light you’re hitting with it, the more you’re going to exacerbate certain aspects of the shading that you may or may not want, like sub-surface scattering. That’s the big hero where we are introducing the audience for the first time to the young Indy. It’s just got to be spot-on perfect.”
“As I said, we were fortunate to have Harrison driving the performance because he gives the performance that you want. It was also perfect reference for our artists, to know you have a ‘feeling’ that’s evoked from watching this right now when we make him 42 years old. You’ve got to feel that same thing without question.”
