A look at photoreal CG faces, AI re-lighting, BBQ steak reference, optical flow and iPad AR tools and a sneaky Texas Switch you may have missed.
In this wide-ranging interview with Wētā FX visual effects supervisor Martin Hill, we dive deep into the studio’s work for Stranger Things 5. This is in addition to befores & afters’ earlier coverage in issue #51 of the magazine on the final series of the show.
Lots of fun detail in here about various Wētā FX tools and techniques, and the artistry behind the work. Let’s dive in!
b&a: Can you discuss the approach to replacing young Will for the Demogorgon and Vecna encounter in the opening episode?
Martin Hill: This scene is set in the season 1 timeline – where Will is around 11 years old and the story is, he disappears for most of the season, which made gathering sufficient good reference a challenge. There was no scan or photo shoot of Noah for season 1, and season 2 reference of his face shape had already changed significantly. We pulled all the season 1 footage, every take, including pre-clapper and post-cut and all the publicity stills of the time to build a solid reference base.
The amount of footage didn’t give us enough fidelity for a full AI face swap approach, with output at 4K with the amount of flexibility the Duffer Brothers wanted, so we made a full 3D digital head. We did during the process use AI transfer as a guide, which helped inform some of the posing, but it was never high enough quality to use directly.
The process for building the digital head was selecting existing season 1 shots or footage leaning into poses that were either similar in performance to the shots we were doing for season 5, or showed a good range of head orientations or face shapes, kind of like building a FACS set from found footage. We then matchmoved and animated the face, iterating and observing the differences with the source material and adjusting. Andrei Coval (senior facial modeller) did a terrific job matching the base model and the poses with the limited information.
We matched the lighting of the shots to verify the shading, with Rob Baldwin (lookdev lead) and Indah Maretha (compositing supervisor) directly adjusting the texture maps live in Nuke in 3D, in a stretched out ‘Brazil’ map, as well as in Mari. This gave us both a model matching the ground truth and some performance that was directly from Noah that we could use.
We filmed the performance double actor Luke Kokotek, with additional witness cameras and a stereo pair of small Basler cameras attached to the main Alexa body, which we could derive solid 3D tracking and also depth information. We replaced the face, and augmented or replaced some of the body with our model.
David Yabu (lead facial motion animator) matched the facial motion for these shots to Luke’s performance studying Will’s performance on season 1, and using some of the verified poses and performance from the test shots. Then embellishing with extra direction from the Duffer Brothers, like heightening the fear in Will’s eyes widening them further.
He’s been in the Upside Down for a while at this point, so we added a digital make-up pass to make him more tired and pale with dark circles under his eyes, as well as adding dirt and wetness, we also gave him a much more dishevelled and wet hair style, drooped down from Will’s distinctive bowl cut.
We’re full frame on his face in the library when the vines attach, and pump the black particles into his mouth. For this we simulated the suction of the pipe and the pressure of the small tendrils compressing into his skin, we added extra veins and darkening and bruising in the subsurface on his face. FX also simulated the tear you see in the final shot, which was a late addition to show poor Will was still conscious.
b&a: For the Demogorgon attack in the Wheeler house in ep 2, what was Wētā FX’s approach in bringing the creatures to life? How did you take any on-set stunt performance further? Was there any bespoke motion capture undertaken? What were the important things to convey about the brutality of a demo?
Martin Hill: Every shot was re-motion captured, then refined in animation. The double was mostly a framing and composition guide for the actors to perform against and cameras to frame to. They held pads for any impact interaction like grabbing Ted’s golf club, though we replaced the club, so his swing performance had force. In the bedroom when the Demo is in the ceiling, it was mostly wirework on Holly and we were animating the Demo back into the grabbing of her leg.
Incidentally there’s a great Texas Switch in the shot where the Demogorgon drops from the ceiling, Nell runs across camera exits frame, when the camera catches up the stunt double on wires is thrown across the room. The Demo performance in this shot is very physical, the landing and the throw both have a lot of follow-through, we motion captured the jump (from a lower height), and used a weighted bag for the grab and throw of Holly into the bookcase. Holly’s wire work was sped up a little for the impact. All the Demogorgon’s acrobatic rolls back up into the ceiling were motion captured back at Wētā FX on a speedrail rig so our performer could do the swing and backwards roll.
It was important to embed the Demo in the scene, there were a lot of dynamic flashing lights in the scene this proved very time consuming to match by hand, so we developed an AI tool, led by Masahiro Teraoka and Robert Byrne, to analyze the lighting fluctuation and direction in the plate, and make an equivalent light rig with cadence, color and exposure of each shot in a user-friendly way. While the results were not 100% perfect, they gave a very close match and nailed the cadence, adjusting the results manually from here by the lighting and comp teams became a far quicker task.
Other small but crucial details to embed the Demo were vital, particularly through the slow, creeping through the house. We paid a lot of attention to the compression and recoil of the carpet under foot, and details like little shards of glass sticking and detaching from the Demo’s foot after he’s smashed the glass shower door.
Once Holly escapes being pulled up into the ceiling, we needed to convey the intent of the Demogorgon’s sole purpose was to hunt down Holly and everything else was secondary. Dave Clayton, our animation supervisor, directed the sinister creeping performance, that showed the thought process of the Demo through just head motion of the actor, with emphasis added afterwards to the petals by hand animation.
This is highlighted when Karen, with her wine, obstructs the Demo and launches into him. The Demo is so focused on Holly he’s taken aback at first and takes him a beat to turn his attention and attack Karen, whilst she goes full mother rage and stabs him repeatedly. By the time the Demo focuses on Karen fully and roars, he has a bottle’s worth of broken glass embedded in his face. This was a good example of the stunt double holding the pad so Cara Buono could really lay into it and perform that astonishingly ferocious attack.
b&a: There’s an interesting slow-motion moment as it attacks Holly’s mother – did that bring any particular challenges, and how did you approach those?
Martin Hill: It’s the same approach there’s just less motion blur to hide the details. We needed to match some of the cloth for the wound and expanding blood. There was an impact on the gown on the plate so we could use some of the cloth rippling. The wounds were all matchmoved and tracked on. With the slow motion, the liquid droplets of the fluid sims of the airbourne blood had to be more accurate with the surfacing.
b&a: Can you break down the specific challenges of realizing Vecna 2.0, in particular the build and what went into realizing his skin and vines?
Martin Hill: Firstly the design, we had fantastic concepts from Michael Maher, he needed to be more powerful and more overcome by the Mind Flayer and the Upside Down. We broadened his shoulders and gave him a stronger physique in general, somewhat to compensate for the negative space we’d be taking out of him, you can see through him, and see his spine through his neck at certain angles. Once we did a first motion tests with the design, we suggested adding padding on Jamie Campbell Bower’s sides to widen his arm’s neutral position so he carried himself with more weight which translated to a more powerful performance.
We also knew he was going to be a lot more charred than the season 4 version; we wanted to make the skin surface look viscerally painful with deep lacerations. At the start of lookdev, we did a couple of BBQs taking meat, adding latex layers for skin depth and slicing and tearing wounds in it, then charring and torching the surfaces. We ran photogrammetry passes and used this as the basis of our geometry and textural detail.
The flowing tendrils needed extra geometric and shading layers, with multiple UV flow sets that motion could be delivered along with twisting properties for the various layers. Areas around the tendrils had warbling displacement that also mimicked flow of tendrils under the skin. This could be directly controlled by the animators to show emotion, we treated it a lot like blood pressure, keeping it contained and subtle when Vecna was in control, and pushing it to extremes when he became more irate. You can really see this when Will took control of his body and screams at Max to run, the tendril flow is pacing through him.
b&a: For the animation of Vecna, how did you tackle translating any on-set or later performance into your digital character?
Martin Hill: We knew straightaway that unlike season 4, Vecna 2.0 was going to be almost fully digital (we augmented the plate inside of mouth and eyes for some shots), but we needed to keep Jamie’s performance. We had a full matchmove of Jamie’s body, that we transferred to the CG, with some performance and weight adjustments and taking into account the different limb lengths particularly when he’s gesturing with his hands.
For the face, Jamie’s intense performance would have been very complex to exactly transfer to a facial rig particularly with all the extra 3D vine detail and moving vines which weren’t present in the onset prosthetic. We matchmoved the head and then transferred all the fine details via an optical flow in-house tool called Thistle, which mapped the nuances back onto the renders seamlessly.
b&a: In particular, can you talk about Vecna’s left arm — how did you approach its animation, alongside accompany CFX work for parts of the arm? Can you talk about how that shot of him spiking a soldier through the head?
Martin Hill: This was one of the first shots we concepted before shooting. There we’re some storyboards of the Vecna’s arm stretching out and stabbing soldiers. Yas Arahori, one of our FX supervisors, prior to us even rigging the arm tried a bunch of motion studies of the attack, stabbing, strangling, wrapping around and gouging his eyes, pretty horrible stuff! One of these tests was pretty much what we ended up going with – there a shocking brutality to it that really punctuates the arrival of our villain.
The rig itself was created by one of our Creatures supervisors Jono Dysart, who needed to solve the problem of an arm that can grow 5-6 meters without looking like it stretches. The main design works by many of the vines unravel from the shoulder with multiple new vines growing in the areas. More vines continue to grow out over the length of the arm to keep a sinister spiky silhouette at all times.
b&a: For the Abyss, can you talk about what real-world environments were filmed, how these were captured/surveyed, and then the build process at Wētā FX?
Martin Hill: Originally, we were going to film the Abyss in Plaza Blanca but it was impractical to bring a full film crew and cranes into the area. It didn’t quite have an open enough area that we needed for the final battle, but the Duffer Brothers loved the look of it. We photogrammetry scanned the whole area via drone and used it as the basis of our terrain, with a few story driven changes, adding the slot canyon Nancy lures the Mind Flayer into, for example.
We then previs’d the whole final battle (inside the Mind Flayer and in the Abyss) and mapped it out for the shoot which was filmed in a quarry in Georgia. The filmmakers had our onset iPad based AR tool, Toro, with which they could view the shots, and the massive scale of the Mind Flayer for eyelines and framing.
The final environment was embellished with a lot of detailing of erosion and rocks falling. Sequence visual effects supervisor Thomas Mouraille oversaw the build and texturing of the massive environment, which special attention to what would need to be destruction ready for FX as the Mind Flayer smashed through it.
b&a: How were cloudscapes and continually moving atmospherics handled in the Abyss?
Martin Hill: The sky was made by our Digital Matte Painting team, led by Jean-Baptiste Verdier. It was a combination of full 3D and a 2½D projected layered setup. Many elements were animated like the swirling atmos around the floating rocks. The wind direction effecting clouds of varying heights, giving dimension was echoed in the motion of the effects dust on the ground level.
The ground dust effects were divided into four main components. Direct impacts that happened in the shot was handled as bespoke simulations in FX. Dust kicked up from the legs would be in the air for minutes, so effects supervisor Mike Chrobak and his FX team ran long sims for pre-canned elements that could be placed by layout in the prior shots impact positions with the correct time offset, where they would be visible or shadow into shot, these we then lit and rendered in the hero render with the environment.
We also had pre-canned volumetric sims, that compositing could place where needed on a shot with the correct wind drift direction, as we were using our volume re-lighting tool Shadowsling, we could apply the same godrays/shadows as everything else in the scene. This tool was a real efficiency for volume renders, if the sun position or the Mind Flayer changed after rendering, the key light could still be moved and the lighting updated for all the rendered volume elements in the deep comp with correct shadowing.
The setup by compositing lead Akshay Khanna and compositing supervisor Indah Maretha was invaluable for being able to visualize and move the lights for volumetric elements in the comp. Lastly for the deep background, Matte Painting had pre-rendered elements on cards.
b&a: For the Mind Flayer, what were some of the earliest kinds of range of movement tests you played with in terms of its walk, its attacking moments, and also its facial animation, mouth opening and screeching motions? What helped give the creature scale?
Martin Hill: The scale of the Mind Flayer starts with the design, when in the Pain Tree form, where Vecna is holding the captured kids, needed to disguise the Mind Flayer in plain sight, with the big reveal only happening when our heroes, seemingly at the end of the journey, approach it. It needed to be immense, with the tips of the legs scraping the sky. The detailing of the model by Layne Howe and James Moore was essential for the scale, sculpting fine layered organic forms that feel like they’ve grown rather and distorted than been carved, making clear distinctions between the different materials of cartilage, bone, vines and shell.
We covered it the Stranger Things language of vines, and thorns which broke up the silhouette, and deposited sand and dust over it to embed it into the environment and give the sense of age and rootedness that belied the moving creature. This gave us interactivity of the dust falling and the vines snapping as the Mind Flayer starts to animate. We referenced the bone snapping from earlier in the series, as the limbs hyper-extend breaking the joints, creating blood waterfalls and pieces of shell breaking off as they double back on themselves.
Once the legs have folded down, you’ll notice about the Mind Flayer is at has 22 legs (at one point it had 26!) all of which need to move in sync without crashing or interpenetrating, the rig for all the ball joints soft tissue with the carapace is quite astonishing!
In terms of motion tests, it was clear it needed to move like a spider (or 2¾ spiders). To achieve this, the animators needed to pay attention to the order of the legs motion, with different gaits having differing orders of leg motion. To show weight and size, each leg impact was embedded into the ground, displacing vast amounts of rock and sand. The Mind Flayer needed to be more solid than the smoke or flesh Mind Flayers of earlier seasons. The creature’s exoskeleton is spider/crab-inspired but not fully rigid, so we can see the secondary motion and slow, heavy jiggle that help illustrate scale, whilst residual vines swing from the body.
Hundreds of articulated teeth and thorns were added and animated to add to the threat when close up to the creature. Animators not only controlled the three-way jaw articulation but also controlled the breath of the Mind Flayer, which in turns drove the simulation of dust and drool of the creature.
b&a: Can you break down what went into that moment of Eleven jumping into the Mind Flayer’s body?
Martin Hill: During previs we set up this completely mismatched face-off between El and the Mind Flayer from the POV of the other kids. We created 2 shots, one wide where El is a speck against the Mind Flayer and another with a long lens that pans from El to the roaring Mind Flayer, and cheated the lighting direction to make these a more iconic silhouette.
El and the Mind Flayer charge, and we worked in the training moves El does earlier in the series to put them into action, then for the final leap she is boosted by the ground being forced up from a leg impact and helping launch El into the air. In an earlier version of the previs during the leap she uses her powers to rip off one of the smaller Mind Flayer’s legs mid-flight, but this unbalanced the power dynamic between the two making El too powerful.
We then set up a camera narrative that implies that El is diving into the mouth/is eaten with a slow motion shot of her stacked against the giant maw of the Mind Flayer, but then switch it up by having El rip open its throat, rearing the monster back to line up the trajectory of El into the wound.
Through these shots there’s a mix of digi-doubles, and relit/retimed plate wire work in here. The wound itself had around 20 layers of skin/fat/viscera that was torn to the sides, simulated by lead FX technical director Marnus Nagel. The sim extended into the following shot from the reverse angle from Pain Tree interior of El’s dramatic entrance ripping Vecna from his vines ready for the showdown.
You can find out more about the visual effects of Stranger Things 5 in issue #51 of the magazine.
