The VFX studio’s history in crafting photoreal CG humans and human-like characters was further extended on the series.
In this further piece in our multi-part coverage of She-Hulk: Attorney at Law, befores & afters chats to Wētā FX visual effects supervisor Guy Williams about his studio’s role in bringing the central character to life.
Working under production visual effects supervisors Janelle Croshaw Ralla, Shannon Justison and Dadi Einarsson, Williams shepherded a team at Wētā FX that shared duties on the CG She-Hulk with Digital Domain. Actor Tatiana Maslany performed the role on set, with Imaginarium handling motion capture duties.
Wētā FX relied on its extensive history in translating a live-action and captured performance into a CG character, utilizing the studio’s proprietary facial animation workflows, machine learning techniques and rendering via Manuka.
Williams breaks down the process, including the meticulous work carried out by Wētā FX to realize She-Hulk’s hair and clothing, especially for…sequins.
A ‘performance partner’
b&a: What methodology did Wētā FX follow, on an overall level, in creating She-Hulk?
Guy Williams: In the beginning we worked with visual effects supervisor Janelle Croshaw Ralla and visual effects producer Brice Parker to figure out how the show was going to be built, how we were going to work with Digital Domain to leverage the best of both houses to create the asset, and how we were going to achieve the result.
Unfortunately, being the height of pandemic, we weren’t able to offer up our teams to go out onto the shoot and be a direct contributor, but we did work with them to help plan and plot out the way we would go about it.
b&a: How did the shoot work?
Guy Williams: In essence, it was in-situ capture. The main thing we all agreed on was that if you have a live action character talking to a digital character, you need to have eyelines for the digital character so that live action actors are looking in the right place. But you also–if you’re doing it smart–put the correct actor in there, so that not only do you have an eyeline, you also have a ‘performance partner’.
One of the best ways you can support actors and help them get the best results is to let them act with other actors, rather than a tennis ball or a big poster card that says, ‘She-Hulk goes here.’ So the question was, how do we capture Tatiana Maslany’s performance in-situ, while at the same time putting her where she belongs.
Tatiana is 5’2″–she’s not the tallest person–and She-Hulk is two full feet taller. We explored your standard options like a backpack rig that puts a head up where the head height should be, or rostrums, or a tennis ball on a stick. At the end of the day, we mostly settled on the path of putting the performer on a platform. You need to have those conversations early because it takes a lot of planning to figure out the actual way to do it on the day.
If She-Hulk walks across the room and stops and talks to a couple of people, you have to have a riser set up that’s two feet tall so Tatiana can walk around the room and look down at people and talk to them. The challenge is, you still want her to walk as much as she needs to without having to step over the riser.
It becomes a bit of a choreographic dance as you need to make sure that you can pull off the individual shots. Something that people sometimes don’t understand is that you shoot movies as slates, not as individual shots. You don’t always shoot little one second long snippets, 30 times. You shoot one 30 second long snippet and cut it up into 30 bite size chunks from different cameras, so you need to make sure that your approach is not going to force them to shoot a lot of slates, as otherwise their shooting schedule could go crazy.
b&a: What was the motion capture approach?
Guy Williams: In terms of capture, this was handled by The Imaginarium Studios. Obviously optical is preferred, but optical also has a very significant setup cost. So on set, production used a lot of Xsens suits. DI4D set things up with a nice stereo helmet rig that gave us everything we needed to be able to run our solver. We also worked with Digital Domain to figure out what the best points set would be so that they didn’t have to carry two point sets for the face.
We asked for the raw data, so we could process all of the raw data on our side. Imaginarium was more than willing to give us processed data for the motion capture, but knowing how our team likes to work, we preferred the raw data from all the motion capture set ups. Our motion edit and mocap teams then processed all of it and gave the animation team the motion that they needed.
The workflow at Wētā FX
b&a: Once it’s all back at Wētā FX, how do you approach the work then? How does machine learning form part of that as well?
Guy Williams: We used the same setup that we used for Gemini Man. We utilised our FDLS (Facial Deep Learning Solver) system, which is our machine learning solver that, with enough training, can figure out the best way to break the dot pattern down into FACS shapes. It takes a little re-tuning every time you use it because you have to train it to the actor. But we trained it to talk to Tatiana’s performance, and to Mark Ruffalo’s performance, so we could get the motion that we needed out of them.
b&a: An interesting side of this is, of course, that you are working on She-Hulk and so is Digital Domain. What are the challenges there to make that as smooth as possible?
Guy Williams: Everybody in the industry knows this, but I find people outside the industry are often surprised to learn that we actually have a very good pipeline for sharing work between facilities. Various companies have led the way and created standards that everybody else has adopted over time. So you have Alembic for passing files back and forth and you have all sorts of different methodologies for sharing different kinds of data sets.
Shading isn’t very shareable, just because everybody’s rendering pipeline is just that little bit different enough that it can screw things up, but textures, models, hair–these are all very easy to share now.
All this stuff was easy to pass back and forth between Digital Domain and us. They started working on the She-Hulk body, got a certain level of approval and handed that off to us. We started doing refinements on it, got approval, and it was handed off back to them—and that process ping ponged back and forth until we ended up with a She-Hulk that everybody loved.
It gets a little bit harder when you get to facial. We obviously share the actual base face shape, but they used a different methodology for their approach. It’s not like we can share FACS shapes, even if we were allowed to. So the facial just came down to the VFX supervisors Dadi and Shannon putting in a lot of effort to make sure they felt the results were similar between the houses.
Skin, hair and delicate art of sequins
b&a: How did you approach She-Hulk’s skin?
Guy Williams: We basically made Tatiana’s skin green. We tried to leverage all of the imperfections in Tatiana’s skin. If you look at somebody’s face, it goes anywhere from a pale blue to a yellow to a red to a pink–all these different tones in a person’s face. You’d think that there’s a ‘flesh’ tone, but there is no such thing as a flesh tone, there are a thousand different tones to your face. So we leveraged all that variation.
The challenge becomes, how do you do various shades of red when you’re talking about green skin, because it’s literally the opposite color. We looked at shades of green going to yellows, which was kind of what was done on Hulk, as well. You can almost reach to red by going through yellow. We played with the saturation levels.
The desire from the filmmakers was to make her look like something out of a glamor magazine, almost like Photoshopped perfection. We ended up going a little bit too far with the initial result and early feedback was that it started to look CG, which obviously it does; people in magazines tend to look CG to some degree. So we got asked to put some of the blemishes back in.
The most beautiful person on the planet has pores. They have moles. They have blemishes. They have skin tone. So all you’re doing is trying to figure out how much and what shape their skin tone, pores, and blemishes are–how much to dial that in.
b&a: She-Hulk’s hair is kind of amazing–what was the approach there?
Guy Williams: When it comes to the hair, we once again started with glamor magazines. The hair presented a unique set of challenges because we wanted 1980s bouffant hair that had modern sensitivities. We wanted really puffed up hair like the comic, but we wanted it to look like more modern hair. We also wanted the hair to flow like a Pantene commercial but with the body of the 1980s hairspray days.
So, right off the bat they wanted her to have this big sort of cowlicky kind of curl on top of her head, this huge sort of volume of hair. The first note we got back was, ‘Well, it doesn’t move’. And I’m like, ‘Well, the amount of hairspray it takes to hold that volume of hair up is prodigious!’
So, you run into the problem of wanting it to look real, but also wanting it to behave in non-real ways, and that can be incredibly hard to achieve. We found the right kind of compromises there, but we also came with some clever approaches. One of the problems was, the more realistic you make hair, the more realistic you have to sculpt it in the first place.
To give you a ‘for instance’, what allows hair to move is basically the stiffness of the rod of the hair. So, you take all the stiffness of the rods out, and the hair will move like silk. It’s gorgeous. The problem you run into rather rapidly is that it holds no form. If you had a curl in the hair, the stiffness is what holds the curl in. You can’t tell it to hold its curl yet still allow it to move. So you have to find the right balance of getting get the curl in there, but then releasing the curl enough so it starts to flow.
The other thing that allows hair to hold its position and also move freely is the friction from one hair to another, sort of surface friction. It’s what allows hair to sit on top of other hair and have a different shape but not slide back. So the stiffness is no longer a factor, but then all of a sudden, the challenge becomes friction.
To try and solve this, you might over-style the groom. If you want a curl, you make the curl even tighter. You make it very tight over to the point where it doesn’t look correct, but then when you relax it, it pulls back out to what you wanted it to be, while still having enough non-stiffness to it. The rods are allowed to be flexible enough that they can move and have some beautiful bounce and fluidity.
Then to solve the friction issue we came up with a couple of different solutions. I kept calling one of them the ‘finger in a light socket’ look, where you sculpt the hair to look good, and then you go back and you take that sculpt and you pull all the hair, so it goes in different directions. It’s almost like they’re in zero gravity.
Instead of her hair hanging down on her shoulders, She-Hulk’s hair is this big ball around her head, but it still has all the proper curls and waves in it. When you turn gravity on, it all then falls back into the final proper groom and looks right. And now because you’ve untangled it, the friction is no longer a problem.
The thing is, if you just groom it into its final result, it’s almost impossible to see what’s happening inside of that groom. What’s actually happening is that some of the hair is pushing through some of the other hair. So, two clumps of hair have started to co-mingle—from the outside it looks fine, but from the inside they’re in collision. The second you turn on dynamics, they’re saying, ‘Well, we’ve got a lot of friction here that we can’t get over’, so they don’t move independently and the whole hair moves.
The result that you end up with is it starts to look like a matted lion’s mane. It has all this beautiful form, but it moves as one big kind of congealed mass. So, by growing it all out independently, then letting it fall back into contact, the collisions do the proper thing. All of a sudden your hair is able to have high friction but not be stuck together, have low rod stiffness but not lose its groom. You’re able to get this absolutely stunning result out of the hair. It takes a bit of time, but the result speaks for itself. It’s what you have to do to get hair to be that realistic.
b&a: Then there’s She-Hulk’s many outfits, how did you tackle these?
Guy Williams: We were lucky, we only had three families of outfits. We had a lot of outfits–don’t get me wrong–we had probably 12 outfits for Jennifer alone, and Hulk had another five or six. But we were lucky in that for the most part She-Hulk wears crop-top T-shirts or tank tops, and a tank top is just a T-shirt without sleeves, so they’re very similar. And then she had casual shorts, casual knit shorts, so it was pretty straightforward for us to do a lot of our clothing simulations. We were able to do very high resolution simulations because the clothes were actually pretty simple. And then all we did was change the collars and the sleeves to get the various permutations of the T-shirts she had to wear and changing the textures.
The one outfit that we had that was very complex was her gala dress. From a simulation standpoint, it’s challenging because it’s not just one layer, the cloth actually has multiple layers. There’s an outside layer where all the sequins are, and then there’s an inside layer, which is a sort of lacy blue fabric. But then she has another layer under that that’s like another layer of lacy blue fabric, a slip almost. And you have to get all of that right or else the clothes don’t wear right.
The other thing that made that kind of challenging was the sequins. They aren’t just a texture. They’re dimensional. They’re a little disc, and you can’t just wrap them onto the surface, because if you do that then they’ll start to bend. The thing that makes a sequin look like a sequin is that it’s a little flat mirror, and the second you bend it, it starts gathering light instead of chattering on and off, depending on the angle that it’s facing.
This meant, for that dress, we had to figure out all the weights. Now, nothing in CG weighs anything, but we know that sequins weigh a lot. We knew the weight of the dress, so we applied that to the fabric, to get the fabric to solve correctly, and then we went back and welded all the sequins on after the bake, so that they could hold their proper shape. They’re basically just bolted on, every sequin, one at a time.
What you end up with is this gorgeous sequin dress with the right result. And then you get into the shading of it all, as you have to model the sequins to the right shape. It’s not just a little disc. It’s almost a flattened cone with a hole in the middle. And on top of that, there were three different sequins on the dress. Some were solid silvers, some were very transparent, some were blue and transparent. They all had different levels of shininess.
Finally, Todd Hulk
b&a: You worked on Hulk and Abomination as well, but I wanted to ask about Todd Hulk–how was that character to work on?
Guy Williams: He was in some ways one of the most fun characters we worked on because he’s just absolutely absurd. The really fun part about it was that the actor did an amazing job of performing the role. We tried to be as absolutely faithful as possible to the result. One of the problems we ran into was that we took the actor and made a Hulk version of him to begin with, and the studio came back and said, ‘No, no, no, no, he’s the villain. He has to look bad. He can’t look so good.’
It turns out that when you take that actor and you make him 7’1″ and give him huge Hulk-like muscles and chisel his face, he turns into an absolute Adonis. He was just way too attractive—that was the challenge we had with Todd. For reference, we had to find bodybuilders who didn’t work on all of their the muscles equally. For example, Todd has massively formed biceps and chest muscles, but a very thin stomach. He hasn’t done a lot of core work. Todd has massive thigh muscles, but very stumpy little peg-like calves, and his head stays small compared to his body.
Then we had to make his face really heavily lean in to the blemishes and blotches so that he just didn’t look too beautiful. We took his rather good haircut and just messed it up even more, as if he had just been rolling around, so that he looked a little bit more maniacal.
The other challenge we had with Todd was we really wanted to milk his transformation. So, instead of him just growing into Todd Hulk, the arm goes and then the feet go and then the other arm goes and the body goes, but the head’s still stuck in. So you get this moment where he almost looks like he’s about to choke because his head’s stuck down inside of his chest because it hasn’t grown yet. We had to make a rig that allowed us to grow the various parts of the Hulk out one part at a time with different sliders, so that we could really control and choreograph the transformation.
