(And Harry Shearer).
Australian virtual production studios Mod and Electric Lens Company recently collaborated to help actor and voice-over artist Harry Shearer perform as Donald Trump as part of the release of his album, ‘The Many Moods Of Donald Trump.’
What was particularly notable about the production, in addition to the deep fake work done to bring Shearer-as-Trump to life, was that the actor performed scenes at his home in Los Angeles in a motion capture suit—but was directed from Australia. Here, Mod’s Michela Ledwidge explains the process.
b&a: How did Mod come on board this project?
Michela Ledwidge: Mod was brought onto the project by Matt Hermans at Electric Lens Company with whom we’ve teamed up for lots of interesting virtual productions and R&D. Matt met Harry Shearer when he was briefly in Sydney to do his shows at the Sydney Opera House. Sydney went into lockdown for COVID just prior so unfortunately Harry’s gigs were cancelled but, fortunately for us, he was looking for folks to realise his idea of performing as Trump in the videos for his new album. While ELC developed the project’s look and CG pipeline, Mod’s brief was to work out “how we can shoot and edit this thing” as a virtual production in LA, from Sydney!
b&a: Tell me about the challenges of orchestrating the recording of mocap remotely – what set-ups had to be in place at both ends to make this possible from a technical point of view? How were face and body captured?
Michela Ledwidge: The overall challenge was dealing with extraordinary logistics constraints imposed by COVID19. Harry would be at home in LA with only one other person allowed in the room, long-time collaborator and DOP Matt Mindlin. The key set-ups were for face and body capture. Harry wore a HMD-mounted iPhone, an Xsens suit and Manus Prime I gloves. We used UE4 to provide a “magic mirror” live preview of the CG Oval Office environment and characters. Epic Games also gave us access to an early beta of their Live Link Face app which is now freely available. A key challenge was ensuring that, while leveraging UE4 for real-time preview and recording, we were not blocked from getting all data into a more traditional post-production pipeline.
Another challenge was managing the on-set workflow for recording takes. I directed the sessions using a custom “Show Control” tool built in TouchDesigner that acted like a universal remote control to many of the key systems in LA. For example, from the one user interface I could monitor a variety of visual and data feeds and hit RECORD to start audio playback and start record on different devices at once. This meant I could monitor live feeds of the iPhone’s FaceAR solve of Harry’s face, virtual camera view of our CG Trump in the Oval Office, and match that to a live action witness can of our real performer.
Mod’s productions typically lean quite heavily on networked systems and automation because it allows our small team to work efficiently but for this project this was even more critical. It was vital that “our LA crew” Matt Mindlin was kept as free as possible to troubleshoot and deal with hands-on tasks around the set.
b&a: How was that mocap data ultimately used in the digital character? Was the facial animation reliant on any specific AI or deep fake techniques? How was this achieved?
Michela Ledwidge: The mocap data was the basis of the final performance you see in the video with some cleanup in Maya. Facial mocap data from the iPhone was converted to FBX via a Houdini Procedural Data Graph pipeline. Body and hands mocap was captured in MVN Animate. The facial animation of the CG Trump character was then used to drive a deep fake model that was trained on footage of real Trump. Credit for the deep fake goes to Matt Hermans who had the idea and worked out how to wrangle DeepFaceLab (most famous for celebrity deep fake viral videos) as a production tool.
b&a: What do you think your experience means in terms of the future creation of remote content like this? What might you do differently next time?
Michela Ledwidge: I’ve been directing virtual teams for decades and prior to COVID I’ve noticed a certain resistance to remote working, especially with regards to traditional on-set production. As an indie studio, we’ve had no choice but to work lean and use every trick in the book for productions. Collaborations-at-a-distance are second nature to us. Now that larger productions and studios have been forced to accept work-from-home and remote production tools I think it will be hard to put the genie back in the bottle. There doesn’t need to be a size-fits-all model for virtual production and there is definitely a sweet spot between having many collaborators in the room and having them spread across the planet.
In terms of what I might do differently next time, we’re always looking to improve what we can do in terms of interactivity. Interactive tools for production and interactive entertainment in homes and location-based entertainment. For a traditional video output like our work with Harry Shearer, we need to continually improve real-time live preview capabilities using mocap camera and character tools. The more nuanced a performance that I can see live on-set, the less clean-up is required afterwards. I’m excited by what indie teams will be able to do with real-time processes. If virtual production brings film/TV one step closer to the immediacy of live theatre then the holy grail, for me, is having the option of finalising performances and pixels on-set.
