Here’s what I learnt at the VFX Bake-Off 2020

A ton of cool things were shown and revealed at the Academy event

Did you know 2,200 VFX artists worked on Cats? Or that Maz Kanata was an on-set live mocap’d puppet in The Rise of Skywalker? Or that Robert De Niro’s eye color was changed in The Irishman?

These are the things you find out at the Oscars Visual Effects Bake-Off.

It’s a wild night: VFX reps from the 10 films shortlisted for consideration for the visual effects Oscar first do a three-minute presentation of their work while a befores and afters reel plays, followed by a 10 minute reel of final VFX shots from the film, then an on-stage discussion.

In the room, at the Academy’s Samuel Goldwyn Theatre, are hundreds of VFX professionals (and fans – the event is open to anyone) seeing things that have often never been shown to the public or discussed until this point. Some of the coolest things relate to how scenes were shot, where practical effects were employed when you thought it was CG, and where CG was used that you thought was practical.

Here’s my breakdown from what I learned on the night about the films under consideration: Alita: Battle Angel, Avengers: Endgame, Captain Marvel, Cats, Gemini Man, The Irishman, The Lion King, 1917, Star Wars: The Rise of Skywalker and Terminator: Dark Fate (the following is the same order as the Bake-Off presentation).

Gemini Man

(Bill Westenhofer, Guy Williams, Paul Story, Mark Hawker)

– The befores & afters reel showcased Weta Digital’s use of an actor puppet to first re-create a digital version of Will Smith before moving to a younger CG version of the actor. The reel also featured the studio’s procedural pore simulation, among other new approaches
– As well as the younger ‘Junior’ CG Smith, a digital 50 year old Will Smith featured strongly in many scenes
– Asked whether the high frame rate meant that things like tracking etc was easier, the comment was ‘not really’. Paint and roto in general on the show was extremely challenging since artists had to remove Smith in a HMC from the plates, and stand-ins
– An interesting note I hadn’t heard before was the use of infrared LEDs for performance capture worn underneath costumes
– That catacombs fight sequence has a whole lot of completely CG Will Smith v Will Smith (including the super slow-mo moment)
– Final sequence with bottles and food breaking – lots of practical FX mixed with CG sims

The Irishman

(Pablo Helman, Leandro Estebecorena, Nelson Sepulveda-Fauser, Stephane Grabli)

– ILM’s new tech – the three-camera system with infrared capture and the Flux software (including some machine learning techniques) that used lighting and texture information to infer facial deformations – was heavily featured in the breakdown reel, as was absence of facial tracking markers
– There was no keyframe animation done on the actors (ie. their performance was preserved)
– Mention made of other studio contributions to de-ageing (including by SSVFX)
– Lots of laughter (and nods of impression) when the reel showed some of the main actors’ bodies being skinnified, part of making them look younger
– Asked what happens when you go into a film with no facial markers or HMC, Pablo Helman joked, ‘Well, you have a bunch of producers throwing up in the corner.’
– The infrared cameras were treated like witness cameras, but they were always of course so close to main film camera, leading Visual Effects Branch Academy Governor John Knoll to make a quip about how the disrespect VFX witness cameras usually receive was reduced here…
– De Niro’s eye color was changed (he also told the filmmakers, ‘You gave me 30 more years on my career!’)

The Lion King

(Robert Legato, Adam Valdez, Andrew R. Jones, Elliot Newman)

– Robert Legato perhaps made the funniest presentation of the night, as he showed off both the virtual production side of the filmmaking, and MPC’s impressive CG character and environment work
– He began with, ‘You have no idea how hard it is to follow The Irishman and not talk like fucking Joe Pesci the whole time’
– Discussing DOP Caleb Deschanel’s role, it was mentioned that ‘when you have a good cameraman, he automatically puts the sun in the right spot’
– ‘This looks really complicated, I don’t know what it is’ – Rob Legato joking about some imagery of MPC’s muscle/fur sims
– ‘We only had adult Simba available, so we de-aged him…’
– There was an interesting discussion about the actual step-by-step process to deliver a virtual environment and animation of the characters with which to plan shots using the real-time tools developed by Magnopus, and the iterations then on that process, and then lighting, then to final at MPC

1917

(Guillaume Rocheron, Greg Butler, Dominic Tuohy)

– The b&a reel showed the vast amount of work done in planning, on set and then in post by MPC to stitch takes into the continuous shots
– Lots of different ways stitching done, including 2D, two-and-half-D, 3D, digi-doubles. Whole No Man’s Land environment re-created, too
– Post done in around 17 weeks, but rehearsals were something like 24 weeks
– ‘When you have Roger Deakins, it’s like ‘Hey, don’t mess up my camera moves’
– For the final trench run, the actor was hit twice but got up and kept going (it was important to get the take because it involved a whole lot of practical explosions), so the crew carried on shooting
– Flare sequence in the village shot at Shepperton involved magnesium flares specially made (with color chosen by Deakins) and SFX team required to make them last 22 seconds. The war-torn building structures were built at certain heights to allow for exactly the right kinds of shadows
– Generally they had to wait for cloudy skies to shoot scenes (good diffusion of light). On one day which didn’t look good for shooting, Deakins predicted there would be a cloud at 3pm. There was, but he still said, ‘Not yet’, and by 4pm the light was perfect for shooting

Alita: Battle Angel

(Richard Hollander, Eric Saindon, Richard Baneham, Michael Cozens)

– This was the first production Weta Digital utilized its actor puppet approach
– Translating original manga to photoreal performance was a strong theme in the discussion. There were also some frank points made about the reaction to the first trailer (where people said the eyes of the character were too big). ‘Jim (Cameron) said F__ the haters. We ended up going bigger and wider.’
– Incredibly complex shoot since it was native stereo (and actress in capture suit and HMC had to be painted out). They also ended up working with very few clean plates
– Great discussion about preserving Rosa Salazar’s original performance as much as possible
– A stand-out part of the befores & afters reel, for me, was the underwater sequence. There’s just something about the underwater look and then the water pouring off the CG character’s hair and face when she surfaces in the ship that looks so great

Avengers: Endgame

(Dan DeLeeuw, Russell Earl, Matt Aitken, Dan Sudick)

– By now, many would have seen the fantastic VFX breakdown reels out there for Endgame, but it’s still incredible to see where some shots begin with in terms of plates (or nothing) and what VFX artists did with the film
– VFX finished with two-and-a-half weeks to spare before release!
– Entire range of work was discussed – from CG Hulk by ILM and Framestore, DD and Weta’s Thanos, and other work by the many vendors
– Some nice discussion about Weta Digital’s AnimFX procedural effects for destroying Avengers headquarters. The studio also had a very choreographed approach to lighting in the final battle to match the mood of the sequence
– Included a look at the practical effects for the flooding of Avengers HQ
– Practical effects team for the film was huge

Captain Marvel

(Christopher Townsend, Janelle Croshaw Ralla, Craig Hammack, Dan Sudick)

– Hugely notable here that additional visual effects supervisor Janelle Croshaw Ralla was the only woman who was part of the shortlisted contenders this year
– Impressive befores & afters reel revealed the vast plates shot of Brie Larson, and where CG took over for things like her hair and costume and the Captain Marvel binary look
– A big challenge for the VFX team was coming up with a look for things like blasters etc that was different to, say, Iron Man
– De-ageing of Samuel L Jackson: new approach removed need for younger body double, significant research done of his previous films, multiple vendors worked on the hundreds of de-ageing shots, always looking to original plate performance
– We heard some fun stories about CG cat Goose, including tests and the reference required for that Tesseract vomit shot
– Christopher Townsend wins for best gag of the night: ‘I figured if you had done anything to do with de-ageing and cats you would get into the Bake-Off – so we did both!’

Cats

(Phil Brennan, Jason Billington, Matt Jacobs, Paul Dimmer)

– No befores & afters reel shown, but the 10 minute reel was great
– The decision to go with CG cats rather than costumes/prosthetics (which were tested) was because it maintained the actor’s performance
– Actors shot in Xsens suits – they had 12 available for use. When there were more performers than that, they stuck with visual markers. Sensors in costumes could also be easily swapped out
– Tools for tracking face and body included optical flow and machine learning, as well as Technicolor’s proprietary Face Lab toolset
– The approach involved subtly blending back into the CG cats the live action hands, eyes, mouths and sometimes nose. Actors had FACS scans
– Every final frame compared to original plate frame
– 76 CG hero characters, 18,000 performances needed precise matchmoving
– 2,200 VFX artists at peak of production. 1765 shots (every frame of the movie)
– There was some discussion about the VFX work continuing after release of film. This was not night and day changes, but mostly things to refine comp and lighting

Star Wars: The Rise of Skywalker

(Roger Guyett, Neal Scanlan, Patrick Tubach, Dominic Tuohy)

– There are 1900 VFX shots in the film. A theme of this presentation was the combo between practically shot, practical FX, puppetry, CG, FX sims and compositing
– Carrie Fisher shots were live action face composited with CG hair, costume and mostly shot motion control to match perspective of original footage (and she was moving a lot of the time). Stand-in on set in costume
– Maz was a live-action puppet, controlled by performer in motion capture suit (data suit and gloves). Additional performers for head dialogue and eyes/facial expressions
– Planet explosion began as real miniature. Final sandcrawler shot was forced perspective miniature
– Insane amounts of CG pyro throughout the film, especially final battle
– The befores and afters reel revealed many practical set pieces – speeders, skiff, X-Wing being raised etc
– Fun to see puppeteering of Babu Frik by greenscreened puppeteers

Terminator: Dark Fate

(Eric Barba, Jeff White, Alex Wang, Neil Corbould)

– The befores and afters reel showed the opening CG head replacement de-ageing work by ILM for Sarah Connor, John Connor and the Terminator in that flashback sequence (I had never seen any behind the scenes for that yet – incredible. It also seemed to have proof of concept tests from scenes in T2)
– A large number of motion bases were made for the film, including the plane sequence, which had a rubberized lining. Even a Humvee was rubberized
– For the earlier freeway chase, one of the cars was a fully-sized radio controlled vehicle operated by a person in a different van
– There was one scene in the plane sequence that required five head replacements – ‘head replacement central’
– For CG versions of the actors, a number of vendors had digi-double shots, including ILM and Digital Domain
– At one point director Tim Miller saw CG actor iterations and thought they needed something extra, so he sent through some deep fake samples. VFX would take the CG performances and run deep fakes on the head replacements – it helped the last 5% of the way with the shots

So, that’s the 10 films under consideration. VFX Branch Academy members will vote to work out the final five nominees for the VFX Oscar.

Who do you think they will be?