An excerpt from issue #34 of befores & afters magazine.
In season 2 of The Last of Us, Joel (Pedro Pascal) and Ellie (Bella Ramsey) are living in Jackson, Wyoming, which has been barricaded against the threat of an infected invasion. Through a combination of events—including the accidental awakening of thousands of infected under the snow by Abby (Kaitlyn Dever)—a horde does ultimately attack Jackson. Here’s a look at Wētā FX’s role in the main horde attack section of the siege, direct from the latest issue of befores & afters magazine.
The siege takes place in blizzard conditions, and this was something Wētā FX visual effects supervisor Nick Epstein knew would form a major challenge for the final sequence, for which the visual effects studio would be adding in their infected characters, large crowds, destruction, gore and other action. Partly that was because the shoot north of Vancouver took place in all kinds of weather, from sunny days to rainstorms and even snow. “With wildly varying shooting conditions, conforming to a very rigid look was pretty challenging,” he says. “[Production visual effects supervisor] Alex Wang and I had a running bet throughout what was worse: rain or sunshine. Both were bad for us. I think in the end we figured out sunshine was actually the worst. We could work with the rain. The camera was a bit more forgiving for the rain, but the weather was pretty bonkers and that also sometimes led to very long shooting days as well. Sometimes we’d start losing light and yet still have some pretty big setups going on. Again, that would cause some challenges later on in terms of conforming that to the daytime feel of the siege.”
Despite the blizzard, the battle takes place during the day. Previously, much of Wētā FX’s CG infected was seen “largely in a high-key nighttime scenario,” shares Epstein. “You could get away with quite a lot more, but here for the siege, you couldn’t get away with anything. As they were shooting, they really were not shying away from, ‘Alright, well, we’ve got 30 stunt actors here. We need about 200 horde rushing towards the camera.’ Alex and I would discuss between takes the optimal numbers of practical horde in order for the best result later on. Sometimes, things happen and you just end up with a shot that’s suddenly needed to be framed closer and the plate horde is further back, so the CG horde just went in front.”
Epstein says what helped with adding in CG characters was the production’s dedication to making everything feel grounded. “They shoot everything and they go to these huge lengths to build the sets and ground everything in a high level of reality. It sets that realistic bar, which I love. Even if you’re doing a full CG shot, there will be a plate for it, and you are held to that level of realism.”
Part of Wētā FX’s role in creating more than 1,000 infected for the siege was to give each and every one of them a distinctive personality, as Wang explores. “These were people, these were civilians, who would have been from all over Wyoming that came to become the infected horde. They were elderly, they were people in their 20s, 30s, they’re teenagers, they’re kids, people that were big, people that are skinny. Craig really wanted it to feel like there was no repetition. Everyone was unique. That is probably the hardest thing with any kind of crowd system.”
“We ended up working on a system that allowed us to change the clothing and the hair simulations on our hero assets and infected assets,” adds Wang. “The most important thing we felt was to showcase uniqueness within the crowd system. So, we had up to 60 individual mocap sessions. We captured almost 600 animation cycles just so that in any given wide shot, we never felt like there was a repetition. If somebody was falling, they weren’t falling in the same way or place. It’s amazing how quickly you can catch those repeats in the performances. It was really important to just make sure that that didn’t happen.”
Wētā FX animation supervisor Dennis Yoo, who orchestrated animation cycles for the horde starting with motion capture sessions carried out on Wētā FX’s capture volume in Wellington, New Zealand, concurs. As he began analyzing what was needed for the horde, he realized that a heightened level of organicness and ‘horde interaction’ was required. Furthermore, the nature of individual runs became important. “I had several performers doing different types of runs, but the problem was that those individuals still look like they’re the same person. Even though they’re intended to be different infected characters, you would look at it and go, ‘Isn’t that the same guy?’ It’s not the same motion, but it is the same guy. You see this mosaic tile effect of motion where you start pointing out the same people. That was also where I just had to go back to the stage and capture more and have different performers because we couldn’t use the same people.”
Yoo also notes that during motion capture sessions, he would look to get performers to act differently in various takes in order to represent different infected. “The problem with that is they can sometimes overact and you start getting a Monty Python kind of run.”
The ultimate approach to animating the infected horde was to start with this base of motion capture (which itself goes through a motion editing stage), with keyframing, cycles and crowd simulation all part of the mix. Keyframe animation was relied upon for hero horde characters as well as for moments where a performer could not perform the specific action in motion capture, such as breaking their neck. “If they’re close up and they’re doing bespoke things, it’s always keyframe,” explains Yoo. “If the motion isn’t changing much and the creators are happy with the motion, it’s usually left with our motion edit department, who works out of the mocap. And then our crowds department uses Massive for the mid-ground or background characters.”
Usually, crowds crafted in Massive do not usually feature heavy cloth simulation, but Wētā FX was keen to push even these crowd agents to have dynamic cloth sims to match the other CG puppets appearing closer to camera. “When you talk about ‘crowd’ work,” relates Epstein, “sometimes it implies there’s been some sort of compromise made. On this show, I honestly feel like we made none. Yes, we have our regular efficiencies in terms of getting things rendered and so on, but we did have, in a lot of cases, more than 500 to 600 agents that are with fully cloth and hair sim going through.”
To that end, every single member of the infected crowd was built to hero level to allow for close-ups and interaction with stunt performers—each infected had their own FACS-based facial motion shapes and fully simulated hair and clothing. “To do that, we ran everything, including shots with 500- plus agents, through our proprietary Loki cloth and hair solver framework,” details Epstein. “The shots just took a bit longer, but essentially we were able to parallelize them on our renderfarm. Wherever you look, you get those really nice profile breaks with bits of cloth bellowing in the wind and hair doing the right thing. And even far away when you’re next to a plate character, our CG characters were doing the same thing, which was great.”
Crowd simulation techniques were also vital for representing dead bodies, including piles of the infected as they attempt to break through Jackson’s protective walls. Wētā FX keyframe animated hero infected as they stumble over the bodies, and then used a combination of Nuance for motion editing and Ragdoll Dynamics simulation on the collection of bodies. “It was a downward shot and it was just tons of dead bodies and we had the whole crowd running over them,” outlines Yoo. “We needed to make them all feel like heavy bodies rather than a sack of hay.”
“I just remember our creatures lead, Claudio Gonzales said, ‘Sorry, you want me to sim what!?’” shares Epstein. “I don’t know how else we would’ve done these mounds of bodies. They were mounds of over 100 infected lying up against the wall, and then we had to have our infected run over the top of them. Obviously, if you don’t have the reaction in the mounds or the reaction from the running over, the whole thing’s going to break. There was quite a lot of back and forth between our animation and creatures teams to make sure that we could cloth solve and hair solve those ragdoll bodies.”
FX simulations from Wētā FX were necessary for snow interaction, fire, charring, destruction, and gore FX for the attack. In particular, the horde needed to run through varying depths of piles of snow. The ground interaction effects were required for both on-set performers that needed to kick up the snow, as well as Wētā FX’s CG infected. “FX supervisor Sumit Pabbi came up with a ground interaction system that was very flexible. It would work in deep snow, say for the Abby chase sequence, and it would also work in hardpack thinner snow like on Main Street and everything in between. The stuff outside the gate as well was variable in terms of how much snow kick-up the clients wanted.” “Sumit would keep us grounded too, in the motion department,” says Yoo, “where we couldn’t fake anything off- camera. If we were faking anything off-camera, it’d explode the snow or if they weren’t touching the snow, our characters, then there would be no kick-up. One of the other things that took up a lot of our time was something that you might not even think about, where you have a crowd having run through snow, which means the snow the other characters run through can’t be fresh snow. We actually had to get the characters to keep running or have a pre-roll of run before that just so that you see that the snow was trampled on. It’s invisible. You wouldn’t even know.”
The studio also developed a gore system related to the infected within Nuke. “This was developed by Owen Longstaff,” notes Epstein. “It would essentially use Nuke particles combined with some spatter elements and spatter sims. You would define trajectories based on incoming trajectories and based on where all the people were shooting from, and it would do things like reveal spatter on the ground over a couple of frames. You could have control over that. Some of my favorite shots were when they come around the corner of Main Street. I think we called it the ‘conveyor belt of doom’, where they just immediately start getting mown down and it’s just gore going everywhere.”
“We had to do a sort of similar thing for all the dead bodies that were piling up,” continues Epstein. “To run simulations on all of them would just be too laborious. So we came up with a projection system where, whether it was a CG corpse or a plate corpse, it didn’t really matter. We’d extract the depth, project back onto that, and then we could control the amount of spatter and the size of the patterns.”
In some shots, members of the horde are also on fire. “We actually had a really great system for that,” says Epstein. “I’m very grateful to Tomohiro Okita, one of our FX leads who came up with this. We got some creative direction on how much and when they should be on fire. They were quite exacting about that, so we built this system that would allow us to tag anyone who should be on fire. It was fully controllable as they’re running through. There was also a mixture of plate fire and CG fire. We had to project the plates back on to have a criteria for when these guys should have been in contact with the fire. So they run through the fire, catch on fire, and then you could change the lifespan. We could vary it based on what the character would end up doing. The system would analyze the motion and go, ‘Oh, this is a run, or this guy’s dead, or this guy’s just smoldering.’ And if the guy was running a certain speed, then obviously the flames would behave a certain way versus if someone’s just flailing around.”
To help fend off the infected once they enter Jackson’s gates, the townspeople deploy dogs, which attack with much success. On set, trained dogs were filmed for shots of them running down the street. When they attack, however, those dogs were CG. “Wētā FX did an amazing job with the CG dogs,” states Wang. “That is some of the best keyframe animation I’ve ever seen. There’s a shot of a dog biting down on the neck of one of the infected, and you could feel it was a real dog when it was just in early phases, where it was not even rendered. If you can sell it in animation, you are honestly over halfway there. The performance is everything.”
The dog visual effects began with scanning the on-set dogs, a task carried out by Clear Angle Studios. “That was probably my favorite day on set,” recalls Epstein. “As a dog lover, I spent way too long with Clear Angle on that day. We had to capture HDRIs and all the texture reference and things like that, too. In terms of what production shot, there’s just a couple of establishing shots where it’s actually a mixture of CG and practical dogs. They went to the trouble of shooting plate after plate, put up a greenscreen on the street, and just had the dogs run towards the camera. Then we had a few real dogs to use to intersperse with our CG ones.”
For the shot of the dog going for an infected’s neck, a stunt performer leaned over in the suitable way without any kind of stand-in or stuffy for the dog. “He just made that up and then we had to conform the dog to that,” says Epstein. “Craig was really happy with the first version of that we showed. It was much more about how much blood should we show, how gory, and how much of the neck wound to show off.”
In terms of animation, Yoo arranged for several animators to keyframe key dog actions and vignettes. “That one scene with the whip pan of the dogs where we hand off to CG dogs was done by Jason Snyman. He would grab the keyframed actions and little vignettes and put them altogether. We threw tons of dogs in there. Just getting them to look real was the harder part.”
Issue #34 is out and covers the practical and digital effects in ‘The Last of Us’ season 2
As the attack continues, a number of the bodies seen lying on the street in the shots were practical ones from Gower’s team. Wētā FX complemented these bodies with CG ones, as well as adding to the bodies and streetscape an extra layer of carnage made up of blood and gore, largely via compositing. A compositing-based approach was also used for the significant amount of snowstorm and related atmospherics in the sequence, as Epstein breaks down.
“We had very variable conditions while filming, and we needed to be able to deal with shots that were filmed with rain, with broad sunshine, no snow, real snow, fake snow. Compositing supervisor Tobias Wiesner came up with a system in Nuke which would leverage some of the machine learning depth and segmentation tools we’ve got and combine that with Nuke particle sims for the snow and pre-generated volume simulations that came from FX. This system would essentially distribute them based on where the camera was, based on an input wind direction, and essentially it gave us full control. The artists in Nuke were able to dial down the amount of snow and interactively see what the weather was going to be.”
“Toby also developed a pretty clever depth to deep re-mapping approach,” adds Epstein. “We would insert the Lidar to get accurate depth information from the camera, and then you’re essentially re-projecting this sometimes flickery machine learning depth pass onto the deep data, giving you a very cohesive and remappable depth map through basically any plate. We’d obviously have to augment that with roto to get to QC-level edges, but it was a pretty flexible toolkit and honestly I’m not sure if we would’ve been so successful without it.”
Grab the full issue of the magazine here, which also includes discussion of Wētā FX’s earlier work on the awakening of the horde, and on making the bloater.
