Industrial Light & Magic’s Todd Vaziri digs into the details on ‘Dungeons & Dragons’ (and also delivers us a VFX masterclass on working with anamorphic plates).
If you don’t already follow Industrial Light & Magic compositing supervisor Todd Vaziri on Twitter, you should do so right now. His musings on visual effects are unmatched in terms of insight and wit (and, it turns out, so are his comments about turning off motion smoothing on your TV and the release of old shows in the wrong aspect ratio).
When I found out I was able to speak to Vaziri about his compositing supervision role on directors Jonathan Goldstein and John Francis Daley’s Dungeons & Dragons: Honor Among Thieves—on which Vaziri worked directly with production visual effects supervisor Ben Snow—I was excited for the chance to get somewhat of a different kind of take on a huge visual effects production than I might normally get.
Which is exactly what happened. Vaziri goes into some super-fun detail here about aspects of visual effects I think we don’t always hear about.
Here’s what we discuss: realizing the proper weight of the creature Owlbear (Doric); replicating action beats with CG characters as if a real camera operator was there on the day, dealing with character transformations; approaching the look and feel of a heavy VFX fantasy movie with something much more grounded in realism; the challenges of anamorphic plates in VFX shot production; and, finally, what proved to be one of the unexpectedly toughest visual effects sequences in the film (and how it involved roto’ing out one character, and then putting him back in).
Getting a large CG character’s weight right
b&a: One of the sequences I remember from the trailer and when I watched the film that I thought was so well executed was that first look at Owlbear attacking and mauling the soldiers. I just wonder if you can talk about early conversations you might have had with Ben Snow about how that sequence should look and play out?
Todd Vaziri: Well, we knew Owlbear was going to be a key creature in the show, not just for its cool transformations, but for this scene, and then the big finale with Sofina. So, a lot of effort was put into its design and animation, how it was all going to work for this movie. With these types of creatures that weigh several tons and are also agile, they’re typically depicted and animated in such a way that yes, they may flip over a car, yes, they may knock down a tree, but they’re moving with such energy and such agility that audiences’ brains kind of ‘break’. They don’t believe that something this big and strong could move that quickly. So the challenge was, how do we properly give this Owlbear a sense of scale and weight and yet have it do some pretty incredible agile moves?
There’s a couple of key ingredients that were in play that allowed that to happen. Not only did Ben Snow work very closely with the directors, with that stylistic edict in their heads the entire time, there was never, ever any direction of, ‘Faster, faster, more, quicker movement, quicker movement, make it more like a blur.’ It was more like what we know of large creatures in the real world, how hippos move, how gorillas move.
With that in mind, we thought, ‘If a creature like that was directable, how would we film this scene?’ And it goes to this choreography with the directors and the stunt performers, some of which were on wires, some of which were flung around. It all started from that point. ILM visual effects supervisor Scott Benza is just an amazing animator who totally gets how to make big, heavy things not only depict strength and power, but also agility and energy with his work on the Transformers movies. Then Kevin Martel, who was our animation supervisor, took all that direction and did such an amazing job.
The way we approached it was, if we could direct this creature to do these movements, and yet it was a little bit of a wild creature, how would we film it? So, the camera moves are not perfect. When Owlbear grabs a soldier in its mouth and flings it over its head and slams it into the ground, you’ll notice that the camera is a little bit behind, as if this has been rehearsed but the operator didn’t know exactly when Owlbear would hit those beats. Those types of things really sell it to the audience if you’re trying to give the movie a sense of tangibility, as if this was filmed in front of real cameras. It’s those types of things that really help.
It’s the same in both that first Owlbear sequence where Doric is freeing the girl, and in the big finale with Sofina. The same tenets apply, you’ll always notice that the camera operator–I’m saying camera operator, whereas it’s usually a virtual camera or the augmentation of real physical photography–but the idea is that the camera operator will have rehearsed the shot but they don’t know exactly when the stunt performer or creature’s going to hit that beat.
Mixing live-action and digi-doubles
b&a: I’m always really interested in a fight sequence like this that requires a fully digital character or creature, but also requires fully digital digi-doubles and an environment that then needs to be mixed in with obvious practical live-action photography. As a compositing supe, what are the ways that you realistically transitioned between live-action elements and CG ones here?
Todd Vaziri: The sequences were really cleverly put together in previsualization. So, we would never have a situation where, when Owlbear attacks a soldier, it would be a real soldier that has to transition to a digital one. It was going to be all or none, or it was going to be just hidden with clever edits. This meant we didn’t have to focus our attention on that. We would just focus our attention on, okay, is the armor on the soldier working properly? Are the way the plates on the soldier when he gets slammed down on the ground working? The trickery was built into how the scene was crafted and edited, and we could just focus on the shots.
As a compositor, this show was a nice relief because it was a plate-based show. Almost every single shot in this movie had a photographic plate of some kind or another. Even if we were going to replace almost the entire frame, it showed how the directors really wanted to frame something in-camera. They wanted to have all of the physical aspects of what it takes to film something, if it’s from boat-to-boat, air-to-air, air to ground, how would we shoot this as a real Hollywood movie? So, we’ve got that kind of aesthetics built into the plates.
The challenge for compositing, then, is, well, how do we sell it? We’ve got this creature, this gigantic creature that is attacking all these soldiers, and how do we sell these dynamic shots? Compositor Eddie Porter did the vast majority of the compositing on that sequence. He did an absolutely amazing job managing all of these things.
For example, the interaction of Owlbear with the ground–Owlbear has these claws, these big heavy hooves when it’s transforming from a horse into the bear itself. The art direction was built-in in terms of what the ground was going to be, which was soft, dark ground, like earth. So every time somebody gets smacked down, every big giant footfall, we had to work out what kind of interactions we were going to put in there. And since we were pretty confident that we could do it all with our stock library that we already had in terms of debris and things getting picked up, we experimented a lot with dust and clumps of dirt.
Sometimes your instinct is, if Owlbear is coming down with all of its weight, smashing into the ground and locking its claws into the ground, well, in order to really sell that event, clouds of dirt should fly up several feet in the air. But we learned very quickly, no, that’s not the way to sell the scale. The way to sell the scale is to actually make them smaller. It makes Owlbear seem bigger. It’s sometimes the opposite of what you think, but it made Owlbear feel bigger.
Eddie did an amazing job just putting in these earthy elements and re-timing, re-colorizing, morphing these elements to work for every single shot. We had some on a particular shot where Owlbear uses its head to smack a soldier out of the way. We had a couple of the feathers fly off, and we did that with stock elements as well.
Also, to sweeten those moves, because in that particular shot, when Owlbear hits the executioner and two other soldiers, the executioner and the first soldier were practical. They were in-camera stunt performers on wires being flung out of the way. The third performer was also practical. What we did, though, was we altered their trajectories to fit the animation. We still wanted to have the base raw physical movements, but we had to do slight speedups and warps to get it to work with Owlbear. I look at the sequence now, and I don’t even think about that stuff. Eddie did such an amazing job on it. It’s super fun stuff. It’s like a bread and butter type of creature plate and ‘how do we make this work compositing’ shot. I love it.
Owlbear: Transform!
b&a: How did you contemplate the Owlbear/Doric transformations? Because they’re super neat and I love morphing as a concept in general, but they had a really different look and feel here, which works so well for the style of the film.
Todd Vaziri: In the opening scene, we have the transformation from the horse to Owlbear, and then Owlbear to Doric in her humanoid form. The way that first horse to Owlbear transformation was conceived was, the audience should be thinking, what on earth is going on with this horse? We start pretty tight on the hooves of the horse and we end up with a pretty tight shot of Owlbear doing a triumphant growl. We are never wide enough to see the whole thing, and I think that works for the scene because it’s an action packed scene.
Cutting to a wide shot and being able to absorb the awesome transformation in front of you just doesn’t seem like the right call for a movie like this. The directors really wanted to have this moment of the audience in total confusion, like, ‘What is going on? Oh, this is some kind of magic. Oh my gosh.’ And it all happens within a tilt up, a relatively tight tilt up.
The way that was accomplished was, they had a real horse on set, and the first few frames of the shot is that real horse. It’s galloping around in place as if something’s happening. We then had an incredibly elaborate horse digital model, plus we had our incredibly elaborate Owlbear model, of course.
For the transformation, we started to look at what parts would transform at what times and we tried to not do any one phase all at once. You start to see a few feathers, you start to see bulging happening, you start to see feathers appear from underneath the horse bridle. Then the mass change actually causes the bridle to snap off. We had a lot of fun timing that moment just right until eventually all the feathers and the beak grew out. It’s one of those things where, as opposed to something like American Werewolf in London or other classic transformations, where you get to soak it all in and enjoy the awe–the awesome spectacle nature of this– this is more like, ‘What is going on?’
But, we didn’t want to belabor the point. We wanted to get right to the action. It’s the same thing with how she turns into humanoid form at the end where Owlbear does a tumble, jumps onto another horse and in mid-air transforms into Doric. That was another case where it had to happen very quickly while Owlbear was in mid-air. We also were very conscious of not wanting to have middle frames look grotesque because some of our tests between Owlbear and Sophia Lillis as Doric looked very weird because that’s just what 50% humanoid, 50% Owlbear will give you. So we had to be really careful not to drift into that world. Luckily it was happening over very few frames.
For the jumping on the horse shot, I composited that one. Kevin Reuter was our look development supervisor and Tom Martinek lit it. In that particular shot, the camera’s orbiting around and we used the horse’s head that we were orbiting around almost as a wipe. Once we emerged from the other side of that head, it was 100% humanoid and completely Sophia’s performance from there on.
Grounded things in reality, even for a magical gelatinous cube
b&a: That leads to another question I wanted to ask you. This is a fantasy film–it’s fantastical–but you’ve already mentioned grounding things in reality. What were some of the things you did in VFX that perhaps matched to the physical cinematography or were aimed at keeping things grounded like that?
Todd Vaziri: Well, and it’s especially hard when a movie, like you said, is so fantastical, with nothing but magical spells from ear to ear and transformations and creatures and everything. But for this particular movie, the directors really wanted to give the impression that you’re just going along with these characters. This is a road film, this is a character ensemble. You’re going along on an adventure. The audience has a strong connection with these characters. They’re flawed, they’re funny, they mess up a lot in this movie, all of which is very human.
So, the action was choreographed so that the camera work felt plausible and tangible, airing on the side of in-camera rather than ‘look at me’ spectacle. Ben Snow was extremely adamant that if there was something that we needed to point out in a wide shot–say, where you needed to see a particular detail–we would have to use the same tricks any on-set cinematographer and production designer would have to do to try to get an audience to look at the right thing, either with lights or staging or blocking.
We would not want to dive into things like hyper saturation, harsh rim lights or just blasting with light, or cameras of God where the choreography of the camera is something that can only happen if it existed in the virtual world. We were going to forget about all that stuff.
It’s especially hard when you’re doing things like magical spells and gelatinous cubes, which is something we’ve never really depicted on film before, yet it’s tangible enough and relatable enough for us to get our audience to understand it. In fact, for the shots of the gelatinous cube, we only departed from this approach once, where we go through the cavity that Doric creates as she transforms into a snake in order to escape. It’s a rare moment where we’re actually piggybacked on top of a snake. But still, even in that shot, we tried to add all of the things that you would expect to see if the camera traveled through gelatin.
Otherwise, the way we approached these shots was–especially when the characters are in the cube and have no control over their bodies and their facial features are completely stretched–the shots had to be funny. It had to look like they were truly shocked to be there. It was a lot of warping and compositing and taking different photographic plates of the actors with Saran wrap placed against their face for a few moments so that we get some bizzaro expressions.
Now, we had seen all of these poses. But what would it look like if we had this gigantic 10 foot by 10 foot gelatin cube? How would it photograph? Ben Snow did a lot of small-scale tests and we realized, yes, we had to build these shots in such a way that was first-off 100% realistic. Then we diverge and we go, okay, well it’s also a story, we have to tell the story, with just enough refraction, just enough reflection of the environment, just enough fogginess and lack of clarity in the gelatin. All of this is in service of, does the audience get what we’re trying to do with the story for this particular moment? We just tried to think of it in terms of, how would anybody on set do this? If you could safely put actors inside a giant jello cube, what would be the smartest best way to tell the story with these cameras?
The art of (and differences between) anamorphic and spherical lenses
b&a: Just in terms of replicating reality again, what about just matching lenses on the film? That is often something we don’t always get to talk about in compositing. I was thinking anamorphic versus spherical which was maybe something you and I talked about a long time ago on Transformers.
Todd Vaziri: Way back on Transformers, that was the most recent resurgence of anamorphic in feature films, because it seems like they were gone for a long time. And ever since that movie and Mission: Impossible III and Star Trek and then Star Wars, I cannot get away from anamorphic!
I did so much study of anamorphic lenses for Mission: Impossible III and Transformers, and I’m literally using some of the same reference when I’m talking to artists about how the edges of frame need to work, how the depth of field reacts differently, the vignette, the channel separation, all those things. I’m still referencing my original body of research.
The directors really wanted this to be an anamorphic picture. For visual effects, everybody gets really nervous because it adds a great deal of complexity on several steps of the process, not the least of which is a layout camera for matchmoving. The anamorphic look is almost, by definition, an abstract destructive look. You’re blurring and warping the frame in a way that is not clinical. But that is inherent to the look, that’s the aesthetic. That’s why certain directors love it, especially where they want their movies to be dreamlike. The lensing has a great deal to do with what kind of tone you want to have for your particular movie.
When we do camera matchmoving, rotoscoping, compositing, and lighting in the computer, ‘clinical’ is good because computers like ones and zeros and 90 degree angles and sharp and straight lines. Well, anamorphic gives you none of that. So, it’s always a challenge whenever we get anamorphic plates for these movies. To me, it results in an incredibly beautiful image, but it requires us to take a look at almost every step of our pipeline and customize it to the lenses that are used.
Also, no two anamorphic lenses are the same! So, Ben Snow, having worked on a lot of anamorphic movies in the past, did a really smart thing. He said for as many VFX shots as we can on this movie, we’re going to shoot with spherical lenses. Almost like in the olden days of film, you would shoot your plates either in VistaVision or Super 35, even if it was an anamorphic movie, to reduce the complexity of the visual effects down the line. And if you shoot spherically, then you have much more top and bottom to deal with and you don’t have all the typical aberrations that anamorphic gives you.
For example, the Owlbear sequence at the beginning, every effects shot was shot spherically, which gives us room to pan around and tilt around in the frame. It gives us more flexibility. You don’t want to do too much because then you get that pan and scan look. But I’ll tell you, our paint department loved it, led by Trevor Hazel, our ILM paint and roto supervisor. It’s always better to have more stuff from which to draw from. Let’s say we have a tilt down and you’re replacing half the frame, well, it’s better if we need more things to clone from when we do our crazy paint-outs.
Everybody was pretty happy about having our visual effects plates shot spherically to give us a nice big, rich negative to work from.
Then what you can do is just take a wide screen cut of the final shot for distribution. But there was a point where I said, we could probably do a little bit better. While we cannot quite 100% emulate the anamorphic aberrations that would happen in these plates, I thought, let’s sneak in some anamorphic characteristics into these shots that are plate based, which is something we typically don’t do. We typically want to preserve the integrity of the photography. We don’t want to mess it up.
Well, the first thing you do when you’re trying to emulate the characteristics of anamorphic lenses is, you mess up the frame! You cheat the depth of field, you make it a little bit shallower, you increase the vignette, you do all these sorts of things.
It’s funny, when I was doing these tests, I would always have plenty of anamorphic material to look at as representation or as reference. First, I would make our spherical shot. I would start by giving it the full anamorphic treatment, even including additional lens distortion, which bows out the image a little bit so you don’t get those straight lines at the edges of the frame, specific to each lens.
I would show Ben tests where it’d be, here’s 100%, and here’s an anamorphic shot right next to it. And we would both look at it and go, ‘Oh, that’s great.’ And then we would A/B my anamorphic test with no treatment. It is remarkable how much it’s like, ‘Whoa, it is a giant change, I don’t know if we should be baking all of this stuff in.’
There’s also the question of how much would the DI process bring to the table? Because part of the whole purpose of DI is evening out all the shots, since we were going to be cutting between anamorphic and these spherical plates. So we ended up not going 100%, but instead just tastefully dialing it in per shot.
It’s something I’m really proud of. I think it helps the movie. It’s an additional little ingredient that will hopefully allow the audience to just think of this as a sequence and not in the back of their minds that subconscious thing of, ‘Okay, here’s the effects shot. Okay, now here’s the non-effects shot.’ It’s one thing less that would trigger that response. I’m glad we got to do those experiments.
‘Not only been completely removed, he’s been completely added back, too.’
b&a: Todd, I reckon on every show that there’s always one particularly tricky shot that stands out amongst others. I’m hoping that there might be a shot like that in the film that you could talk about?
Todd Vaziri: We have this saying in visual effects. You never know what the most complicated shot is going to be in your movie, and it’s invariably not going to be as weird or as complex as you thought it was going to be. And then there’s a lot of shots that you thought were just going to be innocuous one-offs and be done, and they end up dragging on for weeks and weeks or even months and months. You just don’t know.
We did have this one sequence that we thought was going to be hard, which was the Ethereal plane sequence where Simon gets the magic helmet and he puts it on and he meets an old wizard and the world is just changing all around him. He’s basically in frozen time, but not quite a frozen time depiction. It’s more like he’s entered a different plane of existence and the reality as he knows it is slowly decaying into something that he doesn’t know.
We knew this was going to be tough from a design standpoint, from an execution standpoint, but this was one of those where we were right. It was going to be the most complicated, most difficult sequence to execute because we were always looking for a look. There was a lot of artwork, there were a lot of discussions. But, as we’ve discussed, this movie needed to feel tangible. Well, here you’ve got an otherworldly world where everything is falling apart where gravity polarity is changing, where things are disintegrating and reconstituting themselves into something and it’s literally something we’ve never seen before. How do you make that work? How do you make that tangible?
Raul Essig, who was the FX supervisor here at ILM, and I collaborated on the sequence. We did all those shots together. And in the end, I’m really happy how it all turned out.
We thought the first shot of the sequence would be pretty straightforward. What happens is, the character Xenk takes his leave of the group at the beach. It’s this funny scene where he walks in a straight line and he hops right over the rock that was right in his way. Then we cut to Doric and Simon, he picks up the helmet and he’s hesitant to put it on. The camera is dollying backwards with Doric and Simon as they exchange words before he summons up the courage to actually put the helmet on. He puts the helmet on.
The brief was: there’s a momentary flash on the helmet, maybe a little bit of a subtle shockwave for the world to indicate that something weird happened, kind of a blink and you miss it kind of thing, and then we start cutting into the rest of the Ethereal plane sequence.
That first shot is on location, in-camera, and then the subsequent shots were shot against bluescreen so that we could control the environment. But that first shot, we were like, ‘Okay, we can make the helmet blink and we could do a little shockwave.’ Well, we worked on that shot for a long time because not only from all the intricacies of how much is too much with how much the helmet should glow, and the shockwave, what should happen back there, how much of the world should be warping even in this initial shot, what should Doric do? Should Doric just go to freeze frame?
There were a lot of things that were transforming, but in order to keep it tangible, we wanted to keep some things instantly recognizable. There were all these fist-sized smooth rocks all over the place. We started to raise them and not have them change and metamorphosize too much, but be more like a gravity polarity shift where we don’t really know what they’re doing. They start to enter frame, and then by the end of the sequence almost the entire frame is filled with rocks, in addition to everything that’s happening behind them.
We thought, ‘Oh, let’s try to introduce that in the first shot.’ It’s a medium shot of them, but he puts on the helmet and it’s only a few more frames until we cut out of there, but we wanted to make sure to telegraph that this is all going to be happening. So, by the end of the shot, at the very bottom of the frame, some rocks start to appear. But in real world space, at the rate that they’re rising, they couldn’t have come from the ground. They would’ve had to have been just a few inches from the bottom of the frame.
I love that little bit of filmmaking trickery because the only thing that matters is what’s in the frame. And if you sell it, it’s fine. It’s kind of like the Buster Keaton rules of filmmaking. If it’s out of frame, it doesn’t exist. And that’s why Buster Keaton was able to craft so many great gags where all it took was the slight pan of a camera to reveal something, you know, ‘Oh, wouldn’t the characters have known that that was there?’ Of course they would’ve known, but you didn’t know. The audience didn’t know. So the pre-roll for those rocks, they were just a couple inches below the frame. They were already hovering a couple feet off the ground at that rate.
As I say, I just love those kinds of tricks just because a more clinical approach to this would’ve been, ‘Oh, there’s not enough time for the rocks to go from the ground up to…’. Well, it doesn’t matter. Does it sell the shot and does it work?
The other thing is that the camera was dollying backwards and so by the end of the shot, the dolly and the track were completely visible. Well, Trevor Hazel painted that out in like mere seconds. He’s an absolute genius. I looked at that and went, ‘Oh my gosh, I don’t know how long that’s going to take to get done.’ And before I even was making bullet points about how hard it would be, he was already finished with it.
So there was that to paint out. There was the helmet itself which had some very chrome-y reflective parts and details, and every once in a while you would see reflections of the movie camera. So I went in and painted those out. That was super fun.
Also, Xenk was still in the shot, walking back out of focus in the background. Then, the edit changed a little bit. At a certain point, production said, ‘Oh, he’s going to be long gone, so can we please paint him out of the shot?’ It’s not the hardest thing in the world, but these are anamorphic lenses, and he was really back there, and the camera was panning around hitting the edge of frame, so it was a little harder than we thought it was going to be. Both Trevor and I were making still frame patches and moving it and having to do roto of Simon’s hair. But, we were pretty happy with all of that.
Then, the edit changed a little bit more because at some point in some test screening, someone said, ‘Oh, we really should see Xenk back there, but not in the position he was in when it was filmed.’
They then ask me, ‘Todd, is there any way you can take his element and put him further back? He’s already crossed there, but he needs to walk in a straight line.’ I’m like, ‘Oh my gosh, how am I going to do this?’
I said, ‘Okay, I’m going to use the photographic image of out of focus Xenk back there.’ I found what I would consider a good walk cycle for him. It was like three seconds of him walking, it was clearly his gait, his style of walking. I said, ‘Trevor, can you roto those 30 frames and I will figure out a way to stabilize the footage and the roto, and I will cycle it.’ So, I created a little walk cycle.
Then I animated him in 3D space so that he would look like he was walking back there in the frame. So now when you look at that shot, little Xenk back there has not only been completely removed, he’s been completely added back, too. I just love that kind of stuff. It’s all in the service of making the edit flow better. Maybe it never would’ve caught anybody’s eye, but if we can help smooth this out in terms of the continuity of it all, I figure, let’s do it.
I just love that kind of stuff because that’s what we’re here for. And in the pre-digital visual effects days, optically, nobody would’ve wanted to touch that! That would’ve been a real pain in the butt. We have these digital tools, so that kind of stuff is fun. It is totally invisible. It’s not the focus of the shot, but it’s a cool thing and I just love that aspect of it.
