How Imageworks tackled the look and feel of ‘Vivo’.
When visual effects supervisor Karl Herbst (Hotel Transylvania 2, Smallfoot) from Sony Pictures Imageworks was asked to take a look at some early concepts for Vivo, he quickly realized there were a number of different styles that would need to be brought to life by the VFX and animation studio. This included sometimes incorporating a painterly and 2D feel to the 3D animated film, which is now streaming on Netflix.
Luckily, of course, Imageworks has regularly had to deliver a diverse range of looks in recent animated fare. In this interview with befores & afters, Herbst advises what the earliest considerations were for the film from his point of view, how cinematographer Roger Deakins came on board Vivo as a visual consultant, and what some of the specific effects challenges were that Imageworks faced.
b&a: I’m always curious, as a VFX supe on an animated feature, what are the very first things that you need to make sure are happening from Imageworks’ point of view?
Karl Herbst: I think more than anything is building a relationship with the team on the creative side. We really sort of gelled very early, which was great. And that allowed us to take a lot of the concepts. My introduction to the film was right at the end of Smallfoot, I was still finishing Smallfoot and I was asked to come look at this film with the team, with Kirk DeMicco, the director. And he pitched it with Carlos Zaragoza, the production designer, to me.
The first thing that came up, they said, ‘What do you think?’ And I was like, ‘Well, you just described a movie that’s got like eight pipelines in it and eight different looks!’ At the time they really wanted to push the envelope a bit more and each song was going to be a different point of view of every character. That scaled back more for story purposes to a degree.
But those early days, that’s basically what happened, we sit at these round tables, look at images together, talk about, what does the art department need to provide and what can we do as testing and development and kind of meet those things in the middle as we go? I always find the hardest part on an animated feature as a visual effects supervisor is I find myself as the curator of small things that I don’t know whether are going to work until the years later. You’re building all this stuff in advance and you really don’t know how well it’s going to come together until you finally see a shot in the movie and that it all came together. And then we move onto the next sequence and figure that out.
So it was like, ‘Okay, we got the 80% working for indoors. Let’s go figure out the 80% for outdoors.’ And then, ‘Let’s go work on these more fantastical sequences and start figuring those out.’ The 2D look pipeline on that really was one of the biggest projects we had. That had to be thought out very early.
b&a: When you say, ‘Working out the look of the film’ what were the different looks that you thought needed multiple pipelines and what did you settle on?
Karl Herbst: We wanted to make sure that Havana had its own look, Key West would have its own look, etc. Very distinct differences of architectural lighting, design, everything. And then you get to the Everglades and then you get to Miami. If you look at each of those locations they are very different from each other, artistically speaking. So one of the things I did very early is say to the team, ‘Okay, I know you guys want to go on all these diverging paths. But what is going to be the common thread across all of this, that’s going to be our foundation that makes sure everything works together?’
Carlos very early had this thought of, he loved 50’s travel posters and he was using those as a backdrop of, ‘Hey, look at the simplicity in these paintings that they did. Look how they treated that sort of distraction in the background.’ And that’s what drove us to decide, ‘Okay, well, we want to do these painterly backgrounds. Okay. Well, what drives that painterly background? And then how do we use it in every location so that we have, again, a common shooting style across all of these things? So that was, again, what early development was really about that. What are the common foundational things?
The next step was then figuring out, well, what do our worlds look like? Because we don’t want to be realistic, but we want to light realistically. So we got Roger Deakins involved and he’s bringing that aesthetic to things.
We looked at miniatures and movies that are shot as miniatures and what are the things about those we like? Well, it’s the materials that are used, concrete is made into this material for this movie as a miniature. So when we were building Havana, we thought we would do oversized textures and they’re going to have these particular looks to them and we’re going to carry that through to everything in the movie. We had this foundational platform that everything looks like it should live in the same world, even though the paths are diverging from each other, driven by story or driven by look in the art department.
b&a: When you’re building up those looks, does the director want to see some tests to say, ‘Yes, that’s the look’? I mean, final tests, more than early concept art?
Karl Herbst: Well, what’s interesting here really is Kirk and Brandon Jeffords, the co-director, really were more story guys and they very early on were like, ‘We brought Roger in for visual consulting, we trust you guys,’ and basically tasked us to work with them and figure out how to make this work. The real driving force of approving a look came from the art department directly rather than the directors on this film. And yes, we were doing tests all the time.
I hadn’t worked with Carlos before. He and I became fast friends, we wanted to figure out what, where do we gel? I call it the dance. It’s that relationship building of like, ‘Well, what are you going to bring to it versus what you’re going to bring to it? And how do we work together?’
Once we got that dance going, Carlos would throw an image to us. Or the art department threw an image and we would go, ‘Okay, let us have a couple of weeks.’ We would come back with, ‘We hit your image and here’s like five other variants. Which one’s starting to speak to you?’ And I would say the theme on this movie became, we like option number four of the variants, as we would take the next expressions down the path and they would go, ‘We love where that’s going. Now let’s take that and go one more step.’
Once that relationship got going, it carried all the way through production. For example, I’ve never had so many first look files on any movie I’ve ever worked on where, we throw up a sequence and they were like, ‘Go ahead and finish it.’ We had built so much of that foundational language between each other, that it just worked all the way through the film.
b&a: One of the things in the film was a particular ‘depth of field’ look. What are the hard things about achieving an interesting depth of field look in animation?
Karl Herbst: I think the hard thing is the software doesn’t want to do anything other than what we know, right? And we have to teach it to do something or teach an artist how to drive that. And the good news was, we were kind of lucky. We’re building off of what Spider-Verse started. They had built a lot of pipeline for what was a ‘texture bombing’ technique. We actually brought some of the dev people from that group in on the early parts of the project, and said ‘Okay, what did you guys already learn?’ And then of course we wanted our own take on it. So we were building foundationally off of that.
So we would use depth mattes to decide what’s going to be in your plane and what is outside of focal plane in the background, where we’re bringing that brush stroking in. That also became a thing where it’s like, okay, water doesn’t work very well with that concept, because you have strike of surface and depth for refraction.
That’s why if you look at water in the movie, it looks somewhat opaque and painterly by design. So we could do that brush stroking to it directly, thinking that’s the reflected ray going back to wherever is in the distance rather than it’s in the plane of focus. And that just became a look. That came out of, again, early tests where we just had Gabi walking through water in the Everglades and we noticed, ‘Oh, the refractions are breaking all of this. What do we want to do?’ And again, artistically working with the team going, ‘Do you want to fix it and make it look more real? Or do we want to go away from that?’ The decision was we’re going to go away.
And then we started figuring out ways to go away, which again was taking that brush stroking technique and coming up with a new variant of it and going there. But all the brush stroking became a combo of a rendering technique and shaders and also then comp where people could drive that based on, we do almost all of our depth of field in post. So we want to be able to control it, saying, ‘Hey, this shot we actually want really shallow depth of field, we need to bring the brush stroking very strongly into the foreground.’
b&a: Since Roger Deakins was a consultant, in terms of cinematography, what did you feel were the things he wanted, or suggested be implemented? And then what was tough about doing that from the Imageworks pipeline point of view?
Karl Herbst: Well, I think there’s a lot of history in animated features of what I call the storybook look. Nothing can go black and nothing going to be blown out white, right? You can’t go way overexposed and you can’t go away under. And so, art director Andy Harkness early keys fit that very well. That was his intentions. He had a real beautiful color palette throughout the movie. It was just gorgeous. And we were always lifting out of that.
Then when Roger got involved, because he came a little bit later than the early development phase, the conversations very quickly switched to, if you look at a lot of CG features, it’s ‘Why is everything in this room lit and why does everything have to have beautiful exposure? Why can’t we let it go in other places?’
The first shot we finished in the film with Roger was actually the death sequence of Andreas. That actually has a couple of stylizations on top of it. One, that shot that’s the wide where Vivo walks in and is sitting in that little sliver of light by the chair looking up at him. Roger, said, just go darker, darker, darker, just get rid of the whole side of frame, blow out everything of where he’s standing. So his influence was that. And he brought that with a lot of photographs originally and just talked us through why he gravitates towards those types of images. And then Andy started adapting that.
We also did something interesting in this film, key-lighting wise, which we hadn’t done before. We actually made a department that just did key lighting on the early two thirds of the film. And then we broke that up as we went. The reason we did that is we wanted a small group of people who could really carry the look across the film and we did it very early. Rough layout would finish and we would immediately go into key lighting and we would do these kickoffs with Roger of, ‘Here’s the concept art, you’ve already helped with camera. You know the story points, what’s your point of view about where to go?’
And he would walk us through, we would just go through shot by shot and talk through and just do annotations of his thoughts of what was there. And just like everything else, we would take those, do them, and then build on them on the next round of development as we moved down the pipeline. So Roger’s influence really was about, camera obviously with DOP Yong Duk Jhun, and then about what kind of exposures and color—going back to that sequence Andy had a really early concept of, ‘Hey, this is a death sequence. Vivo’s going to catch up to the storyline that Andreas is dead. Wouldn’t it be great if the whole world went to saturated other than Vivo in it.’ Like he’s catching up in the story by that desaturation. So, same thing, we did it full color and then we were designing, what things do we want to block out as being desaturated and where to carry that story-wise through that sequence until he gets to the fountain.
b&a: That only makes me a little bit sad for this reason, which is, I obviously didn’t say it in a cinema and I feel like that would have heightened my experience of that scene.
Karl Herbst: Well, I finished the movie in a theater. I was the only person because of COVID for I would say the last eight months of the film that saw the movie on the big screen. It was kind of fun because I’m seeing it big, I’m seeing it with all the color everybody else can’t see at home. And it’s actually taught, for us, what do we want to do as far as a future color pipeline for the facility, knowing that this remote work environment probably is going to continue. And how do we do that in the future and make sure everybody can see the same images?
b&a: What would you say were the big effects moments in the film you had to deal with?
Karl Herbst: It was funny because my effects supervisor for the film was the same one I had on Smallfoot, Theo Vandernoot. And Theo has been in the industry for a long time, great guy to have in your corner doing development. And when I first was bringing him on I’m like, ‘This is going to be a very boring film for you.’ And it turned out not to be in the end. Everglades was a massive thing. That storm sequence, the kickoff I gave to those guys was I want to see every leaf, every branch, everything moving, this storm has to feel like we basically grabbed reference of microbursts and looked at that. And then again, of course, stylize on top of it, which we did for that sequence.
Water became a very difficult thing to consider, how do we sim water and not go super realistic? Anytime there’s a big splash or a water interaction, how to do that in a way that is not just relying on sim, but what’s the artist’s take? And we pulled a lot of reference from old 2D movies and looked at those as our kind of guiding light of how to do that, which a lot of times is particularly it goes up in the air, but you don’t actually see it hit anything. You just see the droplets touch the surface later, which would be taboo normally and CG that you don’t see the finish of the volume carrying through it, right? So we were doing things like that.
We were looking for all of those kinds of abstractions of stylization that harken back to, again, those 50s posters or 2D movies or miniatures. And just trying to bring all that language together. Clouds became a big part of it too. That sequence where they’re flying in the clouds actually got added very late. It was initially not going to be them flying up and interacting with the clouds. It was in one of the kickoffs for layout where that came up—‘Wouldn’t it be cool if,’ and we were like, ‘Alright, let’s go.’ And so trying to come up with cotton candy looking clouds, that could be heart-shaped all the way into the background and how we were going to develop that. Those were the big effects tasks for the film in general.
Cloth on this film was also big, like Andreas’s pants and his outfit. Joe Moshier was the lead character designer. And we always gave him a hard time about small feet. The feet are tiny on his characters in many cases, and the clothes are very tight in certain aspects and then baggy in others. We didn’t want it to look like it was an inflated suit, but we also didn’t want it to completely collapse. So how do you get that? We were looking back at 50s clothes and how nice those pleats are and how stiff some of those panels are. I don’t know how much starch they would use in the ironing of that. But we were referencing those ideas to try to get the characters for cloth and hair. And then applying that obviously throughout the movie with any character.
b&a: I think the general film goer—and this is a good thing—has no idea, of course, about how much hair and cloth sims still need to be done and that there’s not just push a button to do it.
Karl Herbst: You’re right. Our CFX lead Martin Furness, he’s always like, ‘You never let us just hit the button.’ If I see really flowy hair or really flowy cloth, it always screams realism in a way that does not usually fit the look of our films. So for Vivo, Joe and Carlos and that team also came in and very quickly realized we do not want to see super realistic clothing and hair, where everything was trying to be curated. And that’s why a lot of what we looked at early on were miniatures in Kubo. We were going back and looking at that film a lot and going, ‘How did they handle that?’ We didn’t want the clumpy hair that Kubo had but we were referencing it in a way going, ‘How do we take some of the movement that they were doing and apply it to our language and then build off of it from there?’
b&a: There are some fun 2D moments in the film, too, how did you handle these?
Karl Herbst: I just think the 2D look is a really beautiful set of sequences for this film and the tech again there was, we got to build off of a bit what Spider-Verse had done and then take it in a new direction. I think for a lot of people, they’re going to look at that and not realize all of that was actually done in 3D. We do have sequences with 2D elements in them that we just did with 2D animators, but all this 3D was rendered and comp’ed together, just like any other shots. So those were the ones that I think pushed the boundaries for us on this film in terms of finding a new look and finding new techniques.
More Vivo coverage coming soon on the animation of the film…
