How new developments in the studio’s Masquerade and Charlatan tools were part of its digital human work for She-Hulk: Attorney at Law.
befores & afters’ in-depth coverage of She-Hulk: Attorney at Law begins with this interview with Digital Domain visual effects supervisor Jan Philip Cramer.
Cramer worked with production visual effects supervisors Janelle Croshaw Ralla, Shannon Justison and Dadi Einarsson to bring the on-set performance of actor Tatiana Maslany into the photoreal CG realm of the character She-Hulk. And, like Digital Domain had done with Thanos, the VFX studio collaborated with Wētā FX on both producing shots of the CG character for the show.
Along the way, Digital Domain leveraged its Masquerade and Charlatan toolsets and employed new methods to deal with hair, clothing and even make-up. Indeed, She-Hulk tested the DD crew by simply being one of the studio’s first major female CG characters.
Cramer goes into this aspect in much more depth, right here.
The leap from Thanos
b&a: Let’s go back to the beginning–what were some of the main considerations you had for She-Hulk in terms of what Digital Domain would do?
Phil Cramer: We were fortunate to be the first vendor on board with Marvel in 2020. We worked with visual effects supervisor Janelle Croshaw Ralla to do an initial test shoot. Early on, the goal was, ‘How are we going to do the shoot?’ It was a very ambitious project obviously for an episodic series to have a full CG character.
The goal was to have a Thanos-quality character. So, it was all about, how do you capture all that with the timeframes of an episodic? It’s extremely expensive to have infrared (mocap) cameras everywhere in all the locations. So we wanted to find some sort of a hybrid capture system, which we tested here at Digital Domain. We tried to capture simultaneously with Xsens, Vicon and also the ILM’s fractal suit. Poor Tatiana was wearing three mocap suits, one on top of another.
I should add that this was similar to what we did with Thanos, and it would be a shared asset that all the vendors would be happy with. Like with Thanos, we ended up working closely with Wētā FX again for She-Hulk.
b&a: Where had things moved onto after Thanos in terms of DD’s workflow? You have your Masquerade tool and other tools that you’ve developed over the years. What was the general workflow here that you adopted for She-Hulk coming out of on-set capture?
Phil Cramer: We used our proprietary Masquerade 2.0 platform, and one of the biggest aspects of the new version is that we no longer need to acquire training data. Normally, we need Medusa-based training data – i.e. a very high-end frame-to-frame coherent mesh that we use to train the system for machine learning. From that, we would then recreate the high-res facial performance. Now, however, we’re able to generate it on our own using just the HMC data, plus the scans. We realised our internally-generated data is better than the data we receive. For She-Hulk, we still got Medusa data, but we ended up not using it.
It’s also important to remember that Thanos was a bald, angry man, and Tatiana is a young, bubbly actress with a huge range of emotions, and she portrays them very vividly on her face. It turns out that the difficulty was way higher than what we created for Thanos, because Thanos didn’t need to show many emotions. He just looked great when he was looking stoically at the camera or was angry. But now with She-Hulk, we had to show a full range of emotions and facial expressions, including happy, drunk and giggling, all while talking, and the system has to still maintain and capture this. So our face team, under the leadership of Fabrize Visserot and Ron Miller, had to work a lot on the details to push the facial expressions so far. That was something that we hadn’t anticipated, to be honest, the system was pushed beyond what we’ve seen before.
b&a: The hot thing that Digital Domain has pioneered in many iterations in the last couple years is using some machine learning and AI techniques, and obviously that’s wrapped up in your general tools. What can you say here about how it helped with the final result?
Phil Cramer: So first, we start with gap filling on the marker sets, which is quite critical when you go to these extreme expressions, because you have constant data loss, especially when you have so many shots to do. We had to rely heavily on that, but we still went through the same Thanos process of capturing data, and then up-res’ing it.
We have the HMC with the markers, which ends up being a pretty low resolution representation of Tatiana’s face. Masquerade is able to generate via machine learning a very dense mesh that accurately represents Tatiana’s performance, and in turn, drives the animation of She Hulk.
The animators are able to make adjustments underneath on the performance, but we carry this additional skin delta along for the ride, and you get really high frequency fine wrinkles, deforming very nicely.
We also used our new deepfake system called Charlatan – although we used it more for Tatiana to transform into the She-Hulk than for the actual She-Hulk character. Charlatan was created by our Digital Human Group (DHG), and it is not just for face swapping, it has quite a large array of different options that we can use internally at Digital Domain. We first focused on the transformation scenes to generate a really nice looking Tatiana, which ended up being a combination of a CG asset and Charlatan. We used a baseline render of Tatiana, and then machine learning via Charlatan to up-res and make her look more realistic.
We used this a few times, including the fight between She-Hulk and Titania (Jameela Jamil) in the courtroom to replace the stunt performers’ faces. We then used it for She-Hulk during the transformation. We always ran Charlatan to help with the transition into the plate elements.
We also took all raw plates and resized Tatiana’s face to match She-Hulk’s face. This might sound unimportant, but it was critical. Even though they obviously look very similar, She-Hulk’s face is around 10% larger. To then be able to A/B really well, we had to generate new plates for animation that would put Tatiana’s face in the correct place, at the correct scale, so that it could be used better for references.
b&a: Can I just say, I feel like DD has been doing some of the best ‘3D morphs’ between characters, say the shapeshifters in Captain Marvel, and also here for the transformations. I liked here that often it was done off-screen or kind of hidden. Those transitions were very subtle in some ways.
Phil Cramer: The transformations were meant to be more subtle and elegant, rather than painful and violent. There’s a great hero transformation for Tatiana in episode seven, where she joins a group therapy session in the ‘room of feelings.’ We’re really proud of this transformation, and it utilised everything from Charlatan to Masquerade, it really showed off how nuanced Tatiana’s performance was during her transformation. Our DFX supervisor Eric Kasanowski led the team to get to these beautiful results. The production never actually showed the full transformation in a single shot, but we always worked them as a continuous transformation.
For Digital Domain’s work on Captain Marvel, we actually had the same animation supervisor on She-Hulk, Liz Bernard, and so we built on the same techniques we used in that film. The client-side VFX supervisors also worked on Captain Marvel.
Poise and grace
b&a: I wondered what some of the challenges were of getting She-Hulk’s poise correct or her walk cycle correct? Given she’s tall in Hulk form but needed to be graceful as well. What did you find were some of those animation challenges?
Phil Cramer: These motions were obviously one of the biggest concerns for us. Ideally, you always want to have optical mocap available to get the highest quality of data, but that wasn’t always possible, or practical. Given the number of locations and the time frame, we also relied heavily on Xsens suits.
On top of that, it was most important to protect the plate. So basically, Tatiana would always be on a riser. Whenever you would see She-Hulk in any physical setting, there was a riser going right through it. Everything she walked on had giant platforms to ensure all the eyelines were correct. The problem with that is that she isn’t walking normally, she’s carefully stepping to not fall off something, or she’s choosing her footfalls based on the gaps, things like that. Sometimes she would come off a riser to sit down, and then obviously the whole motion becomes not so great. So this proved to be a great challenge.
She would also wear high heels a lot, and in mocap we don’t often deal with characters who wear high heels. So we had to learn a great deal about this. Early on we would meet at a mocap session and ask ‘how do you walk on a plateau shoe? How do you walk on a high heeled shoe?’ We wanted to be accurate about this, and Liz was a massive asset for us on this show. Along with generally being an amazing animation supervisor, she could relate differently to some of those challenges.
If a high heel walk is done incorrectly, it looks really odd. It loses its grace and elegance, and you constantly have to lock the knees almost completely out, which is super challenging in animation. So we had to develop a new ingest system for high heels versus flat shoes versus plateau.
Tatiana was so awesome in how she portrayed the character. From the first moment on the shoot we just put her in a room with all these different props and asked, ‘okay, how is she going to play this character?’ And she did it even wilder than we thought she would. I was like, ‘Oh my god, okay, this is the character!!!’ I mean, she drinks and she twerks, which are things you might not expect somebody to do for a TV comedy. And that just was her personality showing up in the character.
Clothing and hair and make-up and character traits VFX artists don’t always have to deal with
b&a: She-Hulk’s hair is amazing and I think that a lot of people will be concentrating on just the fact that you had to do a digital human superhero, but the hair itself is just its own challenge and own problem to solve. How did you go about solving it?
Phil Cramer: We had three different hairstyles on our end, and Wētā FX had another two or three. This was a massive challenge, and we worked on it from day one. We were super worried about how her hair would look. Was it going to be the super big hair from some of the comic books? How do we make that realistic? We worked with Wētā FX quite a bit on that.
It was a massive simulation challenge for us. And then the shading was done right up until the very last minute to get the feeling of hair coming through correctly, so that it looked soft enough and didn’t feel oily. We had the biggest CFX team we ever had in any show.
Overall, She-Hulk was a first for Digital Domain in many ways. We had the biggest team size and the most character minutes of any Digital Domain show up to this point.
b&a: What about her clothing? I’ve read that you came up with a system called ShapeWear to handle this.
Phil Cramer: First, we wanted to use muscles, since it’s She-Hulk, and muscles bring a lot of advantages. But then we had to create 12 costumes, and costumes change the base geometry if you really think about it.
Now, on a guy, it isn’t so obvious. But for She-Hulk, there are areas of the body that you don’t really talk about in dailies in front of 70 people or so every day. I mean, private areas are heavily affected by shape wear. For example, we noticed that you need different bras depending on the outfit. We needed a compression bra, or a sports bra, or you name it. Then the muscle system had to interact differently depending on the shape wear that we would apply. The entire behaviour would be different if you wear a compression bra versus no bra, or whatever. CFX supervisor Kim Nielsen, with our muscle group, implemented these systems very effectively, and helped us solve these problems quickly.
It’s the same with the pants. We would have tight fitting leggings, and then loose pyjamas, and we had skirts. And so since now she’s meant to be sexy and interacting with men, the focus shifts, as she’s trying to emphasise these areas. One of the main problems is that each adjustment generates a new base model, and traditionally these would be 12 different models that are getting simulated. So we had to work out a way that let us change these areas on the fly, and then – on top of that – make sure the animation that doesn’t use muscles still matches the look.
These are things you don’t think about early on, but you need to get the animation approved in that costume to simulate that particular costume. So we had the issue that for each of these costumes she would look like a giant in them, because the skirt didn’t have the right fit, or the bra wasn’t quite activated. And then we would show this to Marvel, and it didn’t come across how it should.
In turn, we developed a new system called “ShapeWear.” This system handles the desired look per costume, and also allows us to apply the same look early on in animation. This allowed Marvel to see a preview of the costume prior to simulating. It was very challenging having so many different costumes.
Also, Marvel had each of these costumes physically built. So we had actual big and small versions of these costumes to understand how the fabric behaves, and our simulated versions had to match. Often Tatiana would transform into the She-Hulk while wearing them, which added further complications. CFX supervisor Nathan Fok and the whole CFX team did an amazing job handling this, and Michael Melchiorre, my co-VFX supe, spent a lot of time focusing heavily on consistency for the sims.
b&a: It is interesting isn’t it, in general in the past 10, 15 years, that so many of these super heroes are bulky men or male characters. And if they’re female characters there’s a certain suit that you’ve had to create for them. But I guess you are bringing up that you are effectively making a show where She-Hulk wears business wear, but also wears Spandex, also wears a costume occasionally, also wears things to go out to a nightclub or just to wear at home. And they’re kind of things that visual effects studios haven’t necessarily had to totally deal with yet in superhero films.
Phil Cramer: I think so, and I think even in general, we were thinking how many female driven CG characters are there? I mean, there’s Alita, Neytiri and a few others, but we couldn’t come up with many that have large shot counts. To be honest, we were incredibly proud to work on a show that is groundbreaking in that regard.
We also wanted to make sure that we were respectful in how we were talking about these issues, and that no one felt uncomfortable in the way we’re talking about it. It was important to be hyper respectful to try to understand how that would all work on a woman. We haven’t talked about make-up, but that’s the same, we didn’t realise how many things we had to think about to recreate a woman successfully and respectfully.
b&a: I mean, you can obviously render skin and deal with pores and imperfections, but was there a lot of discussion about how much make-up She-Hulk/Jennifer should wear?
Phil Cramer: Yeah, and again, when you’re a guy, well, I did not know about this stuff, so I started watching The Bachelor and The Kardashians. In fact, early on Marvel referenced Kardashian style make-up, which is this contour-based make-up. It’s a brain bender for visual effects because normally we model the face and then you render it, and the model is how the face is structured really, right? That’s where you drive that. But now with contour makeup, you need to act like there’s additional shading and additional shaping on the face that would make it appear skinnier, but it doesn’t, right? It makes it really interesting in terms of how you get the model approved because you then have to think about, ‘Okay, so yes she looks maybe a tad heavy on the cheeks, but we’re going to now put contour make-up that in turn sucks that back in.’
Then the shading around the eye will again change the feeling of how the model actually looks when you render it in grey shade. On top of that, also the behaviour of the pores and the reflectivity.
The contour make-up behaviour was similar to a normal map. You would apply that, and all of a sudden she looks slightly different, and attractive in a different way. That was something we were struggling with early on, especially how to best apply that and how to explain this to Marvel. You constantly want to show the work, but then you have to represent those aspects early on in the concept phase.
And lipstick! Oh my god. There are so many variations of gloss and matte and what it does to the lip structure. It’s fascinating and our team did an amazing job with all that under the leadership of DFX supervisor Daisuke Nagae.
b&a: You’ve talked about this already a little, but there were certainly fight scenes and She-Hulk performing, but also so much of it was just talking, sometimes with really long frame counts. They’re always the hardest things in visual effect shots because you can’t hide anywhere. I just wonder whether there’s one shot in particular that had that challenge.
Phil Cramer: There is one shot in particular in the ‘room of feelings’ episode. She’s sitting in the chair and starts to self-reflect about how great it is to be She-Hulk, but not for her. And then she even starts tearing up and crying towards the end of that shot. It was close to being two minutes long. It’s incredibly difficult to simulate because everything has to be a continuous thing.
For animation, we broke the shot up into six sub-shots so that we would be able to work it. But to publish that animation would take two days. Any animation note would take two weeks to turn around to see it again out of comp.
We told Marvel, ‘You need to review this shot carefully… as we only had time to show it a few times.’ The biggest thing is when you have lip sync issues and something like that, all bets are off and you get lost. Luckily, the shot was being done at a time in production when everything was firing at all cylinders, and we were mass producing. Masquerade was taking care of that, so there was no lip sync problem. The face looked awesome!
We then did a ton between COMP and FX to get the right feeling of the tearing up. It was important that the timing worked, and that it felt realistic. It’s funny sometimes. These long shots on paper should be the biggest nightmare, but everybody loved it throughout every stage of it. Even from the first blocking animation, Marvel said, ‘Wow, this is so great!!!.’ And we were relieved that no one ever doubted that this was going to work.
More on She-Hulk at befores & afters is coming, including with Wētā FX.
