A celebration of Universal Capture, and what was invented at ESC for the Matrix sequels.
This week, the Wachowskis’ The Matrix Reloaded turns 20 (incredibly, The Matrix Revolutions would be released only six months later back in 2003).
Having already stunned the world with bullet time in 1999’s The Matrix, the sequel’s visual effects team, overseen by John Gaeta, came back with even more advancements for the 2003 films. These advancements were in several areas, including photoreal digital humans.
Firstly for Reloaded’s Burly Brawl, and then in the ‘Superpunch’ in Revolutions, audiences witnessed astonishing close-up digital versions of Keanu Reeves, Hugo Weaving and Lawrence Fishburne thanks to an image-based animation process developed at VFX studio ESC Entertainment that was called Universal Capture.
The digital humans were necessary to help realize the balletic (and bullet-time-like) camera and martial arts moves, where scenes would jump in and out of real photography, stunt scenes and virtual photography. Indeed, this was an even further move into virtual cinematography than had been pioneered with bullet time.
Until that time, digital characters, including digital humans, tended to be animated with muscle deformers or blend shapes, but these did not always capture the fine nuances of human performance. The visual effects team had already innovated in image-based techniques for re-creating film sets and locations on The Matrix and other projects, so they embarked on a similar approach here for Reloaded.
The idea was to craft a synthetic human replica of the required actor (ie. Keanu Reeves) by surrounding the actor with multiple hi-definition cameras, shooting a range of performances, and reconstructing that performance with the computer vision techniques of optical flow and photogrammetry, eventually resulting in a CG human face that could work in any scene. If that sounds like what we might do these days with photogrammetry scans or 4D scans of actors, well, you’re right.
Universal Capture would ultimately receive recognition from the Academy in 2015 via a Technical Achievement Award (presented to George Borshukov, Dan Piponi and Kim Libreri). Alongside the incredible photoreal human capture work, ESC also made major developments in motion capture, lighting and rendering, and hair and cloth simulation.
I personally think the Universal Capture or UCAP achievements by ESC were phenomenal back then, and clearly paved the way for workflows of actor capture into the future. I can only imagine what ESC would have pioneered had they not shut down (at one point they were the VFX vendor on what ultimately became Bryan Singer’s Superman film). befores & afters readers will of course know that Kim Libreri and some other members of the ESC team are now at Epic Games.
Interestingly, the early 2000s were somewhat of an ‘arms race’ in the hope to deliver a convincing digital human in a feature film, and some of the other notable achievements were Disney’s Secret Lab Human Face Project (part of a Gemini Man R&D project that was eventually shown at SIGGRAPH 2002), significant research done by Paul Debevec and his team at USC ICT, and a series of digital humans crafted by Sony Pictures Imageworks for films like Spider-Man 2 and beyond.
I can’t possibly hope to describe the technical side of Universal Capture in this quick look back at 20 years of Reloaded, but here are some suggestions for further reading:
Universal Capture – Image-based Facial Animation for “The Matrix Reloaded”, George Borshukov, Dan Piponi, Oystein Larsen, J.P.Lewis, Christina Tempelaar-Lietz, appeared in ACM SIGGRAPH 2003 Sketches and Applications Program, San Diego, CA: ACM SIGGRAPH, July (2003).
Realistic Human Face Rendering for “The Matrix Reloaded”: George Borshukov and J.P.Lewis, appeared in ACM SIGGRAPH 2003 Sketches and Applications Program, San Diego, CA: ACM SIGGRAPH, July (2003).
Measured BRDF in Film Production – Realistic Cloth Appearance for “The Matrix Reloaded”, George Borshukov, appeared in ACM SIGGRAPH 2003 Sketches and Applications Program, San Diego, CA: ACM SIGGRAPH, July (2003).
Bonus bullet time anecdote
In preparation for a session on the 20th anniversary of The Matrix at FMX 2019 with John Gaeta and Kim Libreri, Gaeta actually told befores & afters about plans the duo had for the next generation of bullet time for the Wachowskis’ Jupiter Ascending, perhaps taking photoreal human capture to the next level. I’ll leave you with this quote, and what might have been…
John Gaeta: Kim and I were going to create a new form of bullet time around Jupiter Ascending, but the studio didn’t want to back the cost of some of the tests. We were going to use motion tracking and Sony digital cameras at the time. They were going to be 4K cameras running at high speed, and tracking 16 X-Games sport-type of events with motion sensors, with special LED lighting systems that were also being programmed to control lighting so that they could be put into virtual environments. It would have been some very, very serious kind of captures.