Not to mention…a character full of snakes, or one made of honeycomb and bees!
When Industrial Light & Magic was first tasked by Jungle Cruise director Jaume Collet-Serra to craft the film’s four cursed conquistadors—Spanish explorers rendered immortal but ‘made up’ of elements of the jungle—there were initially plans to have several more.
“Obviously the ones we ended up with incorporated frogs and mud, bees and honey, snakes and vines and branches,” outlines ILM visual effects supervisor John Knoll. “But there were other ideas we explored, one was a guy who was made out of brightly colored jungle birds.”
“He had a whole nest in his face,” adds fellow ILM visual effects supervisor Chad Wiebe. “He would’ve been an interesting character.”
Ultimately, the four conquistadors settled on would become a complex challenge for ILM’s team, working with production VFX supervisor Jake Morrison and Jim Berney. It involved the filming of real actors on set—somewhat akin to what ILM had done on the Pirates of the Caribbean sequels—while also conducting body and facial capture (a task that saw ILM employ a slimmed-down version of its Flux rig originally developed for The Irishman).
The visual effects studio then built and animated entirely CG versions of the characters for their ‘jungle’ states, complete with character FX animation and sims for elements such as the snakes, bees, dripping honey, frogs, mud and other extremities.
befores & afters sat down with Knoll and Wiebe to explore the details of the making of these wild characters.
b&a: Where did you start with working out how to make these conquistadors?
John Knoll: The idea started with “what does a guy made up of mud and frogs look like?” We went through a whole bunch of artwork, explorations, and different versions of that. Probably the hardest of them all was Aguirre (the bag of snakes guy). I mean, what does that look like, exactly?
Chad Wiebe: I think one of the interesting things, too, is that all the characters evolved over time: both as we worked on them, and as we worked through the artwork. Even before we began to shoot them, they were originally intended to be actors that were augmented with certain elements from the jungle. The snake guy, for example, might have just had a “snake arm”, and you’d see bits of snake through his armor. But I think over time they evolved more and more into fully-realized jungle characters.
So what started out initially as just augmenting actors in costume turned into more of a full-CG approach where the majority of the actual actors were replaced. We tried to keep certain parts, such as eyes and mouths, but at the end of the day, many of our shots were fully-CG characters, and that was quite different from what the original brief was.
John Knoll: Yeah, originally they also made these beautiful suits of armor for them. Partly, as a budgetary concession, they only wanted us to do hands and faces, and we would then retain the armor from the live action. But then as we got further into it, there were a couple of practical issues. One was, I think, that some of the armor was uncomfortable and prevented some of the characters from moving the way they wanted. So then production started asking, “Well, can you do the armor, too…?”
As soon as we started doing some of the armor, it became a question of, “shouldn’t the armor have some type of curse to it as well, and have holes in it, and be a more integral part of the character?” As Chad said, that led to the whole character just being fully-CG.
b&a: What did that therefore mean in terms of on-set capture, or, what could be filmed with those actors on set?
John Knoll: We were trying to follow a lot of the same methodology that we had developed for the Pirates movies, which we felt was best both from a performance standpoint, and for the actors playing the roles on set with all the other performers. It gives camera operators something to frame up on, it gives the DP’s someone to light, and then from there we would work on those plates and put our characters into them. So, we had these terrific actors, all playing the conquistadors in all the shots. And then we tracked their performance, but replaced them with CG characters.
b&a: Did that on-set capture go further than a classic ILM IMocap suit?
John Knoll: We were doing facial capture as well. We had a Flux rig on set that we could be driving the performances from based on what the actors were doing.
b&a: Oh, you mean the infrared Irishman-type approach?
John Knoll: Yes, exactly.
ILM’s Flux system developed on The Irishman involved a three-camera system; the main film camera and two side cameras for infrared facial capture. Along with the proprietary Flux software (including some machine learning techniques), the markerless capture methodology used lighting and texture information to infer facial deformations, driving 3D facial models for the ‘final’ performance.
b&a: Had that been moved forward much since Irishman in terms of any other kind of tech or process?
John Knoll: The hardware was different; more advanced.
Chad Wiebe: The footprint was a big thing, too. The cameras that we used in the rig were a lot smaller. I think on Irishman, that was one of the preventative things, which was having these massive camera rigs that made it kind of cumbersome to move things around and get cameras into confined spaces. This time around—thanks to our facial capture engineers—we had a substantially smaller footprint, with much smaller infrared cameras, which I think made it a lot more versatile for shooting.
John Knoll: They were small and lightweight so that they could go on handheld camera rigs. The Irishman rig had two Alexa minis. So it was pretty hefty, and pretty substantial. So we wanted to make it more filmmaker-friendly for handheld and lightweight rigs, and we needed to be able to put it on the Technocrane.
b&a: Although there’d been a costume exploration for these characters, what did you find worked on set in terms of how much make-up they should wear, or hats, or even parts of honey wax, or anything like that? Or was it, nothing?
John Knoll: Well, nothing in terms of that, like the honey wax. We were trying to follow through on them wearing the armor as much as they could, except when there were issues like when we had to do a cable pull, which meant the actor couldn’t wear the chest plate. For the most part, we replaced those armor elements, but they were a fantastic reference. There was a lot of value in having them on the character in the light, because we would study that as we were lighting the faces.
Chad Wiebe: For the actors, too, it really allowed them to be more in the set, and more in the moment. I think a lot of actors prefer to be in that tangible world as opposed to wearing the typical tracking suit, which tends to pull you out of that environment a bit. So, in addition to the great lighting reference for us, it was just great for the actors to be able to actually be there in costume feeling and looking like conquistadors. I think it really helped.
John Knoll: I think it also constrained their movement in ways that help us. If they’re not wearing the armor they’re probably moving their shoulders in ways that you can’t really do.
b&a: Because a few gags happened with these characters like snakes coming out, was there any kind of puppeteering, or stuffies, or anything else on set, that helped with that?
John Knoll: No, I mean, if you see some of the plates, they’re pretty hilarious. There’s one of Puka Michuna getting a snake in the face. So Aguirre, played by actor Edgar Ramirez, does this hand gesture to throw a snake out at Puka, and he reacts with this startled “Aargh!”, shaking and twitching his head. It’s pretty funny.
b&a: What was your workflow taking those plates and going into VFX?
Chad Wiebe: We would start with matchmove and layout. We would immediately start doing the facial capture because that was obviously the most important aspect of each character, and also the most time intensive.
There were a lot of instances where we did augment or change the performance as well. Each character had a lot of different body parts that weren’t necessarily human. So making sure that we can convey the snake guy as very serpentine, and the bee guy as very quirky. Gonzalo—the roots and vines and branches guy—we had an actor for the performances and for the eyelines, but his entire body became about the growth of vines, and leaves, and branches, as well as the foliage growing. So none of his performance was actually used aside from some facial capture, which even then was very limited. He literally only had a small section of human-ish skin left on his face. So we would just completely keep him keyframed for the most part.
John Knoll: Every one of these characters had a few layers of complexity to them, because while they themselves are the character, they’re also forty-five snakes that all have to be animated, and all have to fit inside the envelope. And then a skin sim goes over the top of that because it’s supposed to look a little bit like a human-shaped bag of snakes, kind of shrink-wrapped if you will, bulging and moving with the emotion of the snakes.
Each character has similar challenges. Like the bee guy. He’s the bee guy, but he’s also a swarm of bees that are moving around on the surface, and then flocking around him and Gonzalo (the vines and tree guy), who’s growing, and pieces of him are breaking off, and so there’s all that plant growth. And then the mud guy: he’s constantly dripping mud, but there’s also twigs and leaves and frogs in there.
b&a: Was any of the actual live action plate of the characters’ faces retained?
John Knoll: I think, in the end, they were all replaced.
Chad Wiebe: Right, they were all replaced. The only thing that we were able to keep for certain shots would be some things like eyes and mouths, and we also tried to retain the hair if we could. But in the end there was so much art direction for each shot that we really ended up replacing the majority of them. Again, Aguirre is a good case in point where, because of the fact that his entire face is constantly moving and writhing, and these snakes are moving under this translucent layer of skin, it was then that we had to take it over.
Towards the latter sequences, like the end battle in the Tree of Life environment, a lot of that was shot during reshoots. Some of the dialogue was even switched around, so it was really like a mix-and-match of different performances and different dialogue takes. So, in the end, a lot of those shots ended up as our most intensive fully-digital characters. Those are some of the coolest shots. That was the most fun for us to work on.
b&a: I was curious, then, whether you found that there were very specific challenges with getting the skin quality right, especially in the lush jungle and especially with some very long shots?
Chad Wiebe: Well, one of the shots of the conquistadors by the river is actually our longest-running shot on the show. It was one of our dev shots, where all four of them form at the same time as the camera dollies back, and they’re talking, and there’s dialogue throughout all this. That was our test bed for each of the characters, where we had to ask the questions: “What do these characters look like? How do they grow? How do they form themselves?” So each of them were their own separate dev project that we started very early on, and we just continued with as the characters evolved, and changed, and grew further into the CG realm — as opposed to what was originally slated as being potentially just augmentation.
John Knoll: Originally, too, the idea behind the shot was each character reforms from the jungle and becomes our live-action performer and then they were all in their wardrobe, but they were so much more fun when they were only partially formed, so we just kept thinking: “Look, let’s make the completion of the transition later… and later and later… well, what if they never complete? What if we only get them to 60% by the end of the shot?” And it was just so much more fun. Like Aguirre, you’re just seeing all the holes through him. The bee guy, Sancho, he’s still got large sections of negative space and he’s just cooler that way.
b&a: I’m always interested in the mix of character animation, say here for moving snakes and bees and frogs, and the addition of character effects, CFX, like mud dripping, honey dripping. How did you manage that process as the actual character animation of the characters continued?
Chad Wiebe: That definitely was one of the biggest challenges. And again, Aguirre is a good case in point. As the main conquistador he was also the most complex conquistador, both in terms of just character overhead with the sheer volume of snakes, and then just the different layers and components that had to be added after the fact. For example, we had to take the entire volume-of-a-human and fill it with snakes, writhing and twisting around each other, and not interpenetrating with each other. So we basically had to design a crowd system for that component because it was just way too many snakes for an animator to do.
We had a base layer of crowd-simulated snakes that we called our “tube snakes” and they were just basically snakes that were on predefined shapes within this volume, going around in loops inside. Then, on top of that, animators would have the ability to hand-select any snakes that were within that system, and they could convert those snakes to be “hero snakes”.
And what that would do is import our hero rig—a much heavier and more detailed rig—that allowed for a lot of the intricate motions and the facial animation and just all the complexity that comes along with a twisting, “serpentine” creature. They could swap out as many as they want. And so, for example, when we have a hole in the armor on his shoulder, and you get a nice moment where you see a head or a tail slither by, we would swap that tube snake out with a hero snake to really drive the performance.
In the end we probably have at least one or two dozen hero snakes. We’d often have snakes on top of the armor going around Aguirre’s neck, some of them coming up from branches and on the ground, which would then combine with them.
So it was definitely a very complex setup for our snake management. From there it has to go through our creature dev department where we do the skin sims, cloth sims, hair sims, and armour sims. And all of those elements had to work seamlessly with each other.
b&a: John, you mentioned Pirates of the Caribbean; there’s obviously a real line that can be drawn from that film to these characters—how does the approach then compare to this?
John Knoll: It’s similar, it had some of the same challenges. Some of our Davy Jones characters then were composite characters where there’d be moray eels, and crabs, and other little things that were part of them. And so, this is a step up in complexity where each character was actually dozens of sub-characters, along with their own layers of simulation. It was certainly stylistically meant to evoke the same idea: these stylized, cursed characters, that are individually “themed” by their curse.
Chad Wiebe: You know, they all had their own unique set of challenges and problems and hurdles that you had to overcome to get the point of each character across. But I think at the end of the day, they all really fit the vision of the characters, and each of them had their own quirky, unique attributes that I think made them very visually interesting. And that’s always a rewarding thing to see.
John Knoll: And when we were first embarked on the design process on those characters, it was easy to make the leap of imagination that a guy who’s supposed to be a bag of snakes could be a pretty spooky and memorable character. And then when the brief was, “I want to do a character that’s made from honeycombs and bees”, I was a little unsure how scary or how cool a character like that could end up being. But the art was pretty neat and then he ultimately ended up being, I think, my favorite character in the end.
All images © 2021 Walt Disney Pictures. Courtesy of ILM.
