A celebration of Universal Capture, and what was invented at ESC for the Matrix sequels.
This week, the Wachowskis’ The Matrix Reloaded turns 20 (incredibly, The Matrix Revolutions would be released only six months later back in 2003).
Having already stunned the world with bullet time in 1999’s The Matrix, the sequel’s visual effects team, overseen by John Gaeta, came back with even more advancements for the 2003 films. These advancements were in several areas, including photoreal digital humans.
Firstly for Reloaded’s Burly Brawl, and then in the ‘Superpunch’ in Revolutions, audiences witnessed astonishing close-up digital versions of Keanu Reeves, Hugo Weaving and Lawrence Fishburne thanks to an image-based animation process developed at VFX studio ESC Entertainment that was called Universal Capture.
The digital humans were necessary to help realize the balletic (and bullet-time-like) camera and martial arts moves, where scenes would jump in and out of real photography, stunt scenes and virtual photography. Indeed, this was an even further move into virtual cinematography than had been pioneered with bullet time.
Until that time, digital characters, including digital humans, tended to be animated with muscle deformers or blend shapes, but these did not always capture the fine nuances of human performance. The visual effects team had already innovated in image-based techniques for re-creating film sets and locations on The Matrix and other projects, so they embarked on a similar approach here for Reloaded.
The idea was to craft a synthetic human replica of the required actor (ie. Keanu Reeves) by surrounding the actor with multiple hi-definition cameras, shooting a range of performances, and reconstructing that performance with the computer vision techniques of optical flow and photogrammetry, eventually resulting in a CG human face that could work in any scene. If that sounds like what we might do these days with photogrammetry scans or 4D scans of actors, well, you’re right.
Universal Capture would ultimately receive recognition from the Academy in 2015 via a Technical Achievement Award (presented to George Borshukov, Dan Piponi and Kim Libreri). Alongside the incredible photoreal human capture work, ESC also made major developments in motion capture, lighting and rendering, and hair and cloth simulation.
I personally think the Universal Capture or UCAP achievements by ESC were phenomenal back then, and clearly paved the way for workflows of actor capture into the future. I can only imagine what ESC would have pioneered had they not shut down (at one point they were the VFX vendor on what ultimately became Bryan Singer’s Superman film). befores & afters readers will of course know that Kim Libreri and some other members of the ESC team are now at Epic Games.
Interestingly, the early 2000s were somewhat of an ‘arms race’ in the hope to deliver a convincing digital human in a feature film, and some of the other notable achievements were Disney’s Secret Lab Human Face Project (part of a Gemini Man R&D project that was eventually shown at SIGGRAPH 2002), significant research done by Paul Debevec and his team at USC ICT, and a series of digital humans crafted by Sony Pictures Imageworks for films like Spider-Man 2 and beyond.
I can’t possibly hope to describe the technical side of Universal Capture in this quick look back at 20 years of Reloaded, but here are some suggestions for further reading:
Universal Capture – Image-based Facial Animation for “The Matrix Reloaded”, George Borshukov, Dan Piponi, Oystein Larsen, J.P.Lewis, Christina Tempelaar-Lietz, appeared in ACM SIGGRAPH 2003 Sketches and Applications Program, San Diego, CA: ACM SIGGRAPH, July (2003).
Realistic Human Face Rendering for “The Matrix Reloaded”: George Borshukov and J.P.Lewis, appeared in ACM SIGGRAPH 2003 Sketches and Applications Program, San Diego, CA: ACM SIGGRAPH, July (2003).
Measured BRDF in Film Production – Realistic Cloth Appearance for “The Matrix Reloaded”, George Borshukov, appeared in ACM SIGGRAPH 2003 Sketches and Applications Program, San Diego, CA: ACM SIGGRAPH, July (2003).
Siggraph 2005 Digital Face Cloning Course
Bonus bullet time anecdote
In preparation for a session on the 20th anniversary of The Matrix at FMX 2019 with John Gaeta and Kim Libreri, Gaeta actually told befores & afters about plans the duo had for the next generation of bullet time for the Wachowskis’ Jupiter Ascending, perhaps taking photoreal human capture to the next level. I’ll leave you with this quote, and what might have been…
John Gaeta: Kim and I were going to create a new form of bullet time around Jupiter Ascending, but the studio didn’t want to back the cost of some of the tests. We were going to use motion tracking and Sony digital cameras at the time. They were going to be 4K cameras running at high speed, and tracking 16 X-Games sport-type of events with motion sensors, with special LED lighting systems that were also being programmed to control lighting so that they could be put into virtual environments. It would have been some very, very serious kind of captures.
I always saw this ‘experiment’ of being like an operation where it is deemed a success but the patient still died. From the first time I saw a clip of the Burly Brawl (on Entertainment Tonight I think), I was flabbergasted at how ‘animatic’ and plasticky it all looked, and then in the theater when I realized those were finals, it did a lot to sabotage the moviegoing experience for me (was disappointed with a lot more than the VFX, but this was a biggie … it was like what worked for virtual backgrounds in the first one got put front and center here, yet the reason the virtual stuff worked in the first one is BECAUSE it was background. Similar to how the bluescreen in SW worked because it was against black but you try that against snow backgrounds in EMPIRE and it becomes almost STARCRASH or MESSAGE FROM SPACE looking.
RELOADED may not have been as howlingly awful looking as the Rock in MUMMY RETURNS, but it seemed that bad, owing to such expectations, at least on my end. (but what do I know, whenever I see the flyover of the ship in TITANIC, I’m always seeing how jittery the mocapped people look and it takes me back to the first sail barge shot in JEDI, which for a time was tied for my most hated FX shot ever, up there with the little guy fleeing epsilon IX in ST-TMP.)
I agree with what your wrote Kevin. IMO the demo videos of the faces rotating look very good, passable to me, so it makes me wonder why was it so flat and “CG” looking in the final film. I guess time and money was a factor, would be nice if someone who worked on the film could chime in.
I love TITANIC, but every time I see that amazing shot now, the 3d people really stand out to me, which is shame as the rest of the VFX holds up nicely. Considering JC has already made a few alterations to the film, I would be totally okay with an updated version, that just fixes the old plastic looking CGI.
I think one issue (and one that many have discovered) is that you can have fantastic models with very detailed surfaces that look great close up but as you zoom out you need to handle that detail correctly. The obvious thing is just to do some kind of averaging, which is what you do with images when you zoom out. That leaves you with a flat looking surface. Detailed surfaces scatter light in many directions and this ought to be visible even as you zoom out. So any detail you sacrifice in the surface as you zoom out needs to be replaced with increasing the diversity of ray scattering in the shader. I think there are papers on how to do this now but I don’t think we had a good solution at the time. (Disclaimer: I worked on the animation but not directly on the shading.)
Thanks for writing this, Dan.