Plus a look at how ILM dealt with a moving camera, eyes, and body tracking markers
I talked to ILM visual effects supervisor Pablo Helman previously about The Irishman, where a brand new three camera set-up and Flux software was utilized to capture the actors on set and then enable their likenesses to be re-targeted to a digital version of a younger actor. This was done without any facial markers.
To add to that the story, I wanted to know about three extra specific aspects of ILM’s work in pulling off the effect: how they dealt with a moving camera, how they dealt with the actors’ eyes, and what those cubed tracking markers seen on the actors’ clothing were all about.
Helman explains each aspect below, while ILM also shared with befores & afters these exclusive animated gifs. The gifs show the geometry created from lighting and texture information captured with the three cameras. The shaded render is generated by the Flux software as a representation of the capture on the contemporary digital double of the actor. Later, the capture is retargeted to the appropriate digital younger actors’ age.
A moving camera
Pablo Helman (visual effects supervisor, The Irishman): Camera moves, motion blur and object occlusion are all challenges that Flux has to deal with. Fortunately, the three camera rig helps with all of those. The two witness infrared Alexa minis (to the left and right of the center/director camera) were running at higher FPS and shutter, which minimized motion blur and gave us a very sharp image to supplement the computation. The infrared cameras also helped with occlusion by providing alternative angles of the occluded areas. Another tool available in Flux for motion blur and occlusion is the implementation of roto curves to create areas of interest.
Dealing with eyes
The eyes were a combination of plate and 3D renders. Tracking of the eyes was important because their movement drives wrinkle and micro-wrinkle generation in eyelids and eyebags. As much as possible, we tried to retain the eye reflections from the plate, especially on extreme close up work because you could see the other actors and environment in them. Every shot of De Niro was delivered with an eye matte to the DI for color management. Additionally, we did some remodeling for the eyes (separating the pieces for the sclera and cornea) for this show. With the sclera as an enclosed surface, light scattering yielded better results especially when lit from the side.
Those body tracking markers
Body tracking markers were used for line ups and rigid tracking of body, especially around the shoulders and upper torso. They were used to expedite the layout process especially because the markers show up in three cameras for verification.
