Behind the brushstroke, inkline and frame rate tools made for ‘Across the Spider-Verse’, plus case studies on bringing the hugely complex Mumbattan city and parchment paper-based Vulture to life.
With so many unique styles of characters and worlds in Spider-Man: Across the Spider-Verse, Sony Pictures Animation and Sony Pictures Imageworks relied on its full arsenal of artists and innovative techniques to help craft the film.
Several new tools were created for Across the Spider-Verse, including ones aiding in inklines, brush strokes and frame rates. While these were aspects that had already been tackled to some degree on the first movie, the sequel posed many massive new challenges.
“It was things like seeing actual paint strokes, seeing line work move, and figuring out looks for six different universes,” the film’s visual effects supervisor Michael Lasker tells befores & afters. “It was really a completely different ballgame.”
“We built a lot of things completely from scratch,” adds Lasker. “There was very little we really could reuse from past projects. We certainly built on what we developed in the first movie from Miles’ look, and we built on some things from The Mitchells vs. The Machines, but in general, almost everything was developed from scratch.”
StepSets: dealing with variable frame rates down the line
The first Spider-Verse and now its sequel take full advantage of varying animation frame rates. Sometimes animation is on 1s, sometimes 2s or 3s, right up to 6s. The Spider-Punk character Hobie even has different frame rates for different parts of his body and accessories (like his guitar).
But what comes with animating on different frame rates in Maya can add challenges down the line when scenes are published and move into CFX or need to be combined with Houdini simulations, for example. Allowances at these later stages need to be made for the varied frame rates in order to achieve the desired look in cloth sims or explosions.
To make that process smoother, Imageworks developed what it calls StepSets. StepSets allows for tools down the line to republish the scene on different frame rates and generate FX and sims that way.
“Generally,” says Lasker, “cloth and hair is typically going to follow what animation has done because it all has to sync up, and we’ve gotten pretty good at that. But on the FX end is where it got a little tricky. And then with characters like Punk, we pushed even more onto 4s and 6s, making it really choppy and splitting limbs off so they could animate at different rates, and change costumes at different rates.”
Kismet for linework
Inklines and outlining were things Imageworks dealt with in impressive creative ways on Into the Spider-Verse, but needed to find ways to go even further on this new film, while also retaining a ‘hand-drawn’, sometimes even messy, look and feel. The linework, too, had to update as a character moved and be in sync with any changing animation frame rates.
“On the first Spider-Verse,” explains Lasker, “we had created inklines on characters through animation and through FX. The backgrounds were not heavy on line work. In Mitchells, we did a lot of line work, but again, it wasn’t overly relied upon. For this new movie, we needed line work everywhere, different styles of lines, clean, architectural, messy, sketchy, painterly, everything.”
“Our effects supervisor Pav Grochola oversaw the building of a new tool called Kismet in Houdini. We even had an ‘inking’ team. This was the first CG animated movie I’ve ever heard of that actually had a dedicated inking team of artists that would go in and run inklines and dial inklines in sequences. With this tool, we could make it sketchy, we could make it clean, we could use it for shadows. We could attach it to lights so when they cast a shadow, the shadow would be filled with line work or markers. Some of it also integrated with the Stroke System [discussed below].”
Stroke System: controlling brush strokes
Imageworks made several brush systems for the film. One of these, the Stroke System, was built inside Nuke and worked as a node-based tool that could render brushes in 3D space. Lasker says this tool, and the other brush tools mentioned below, came about after Imageworks saw the very painterly production art department concept explorations for the different world looks.
“We had to build new tools that would allow the artists to fill up these scenes with brush strokes, so one thing we built was this procedural brush stroking system, the Stroke System, and it would allow us to use different kinds of brushes and layer them on objects.”
The brushstrokes were used heavily for Gwen’s Earth-65 world (which actually changes constantly, almost like a mood ring), and Mumbattan, and could be used to represent anything from the beam from a headlight to a cloud of smoke.
“We built the tool so that we could physically paint brushes on objects because we quickly learned that we needed to have brushstrokes move along the form for it to look right,” continues Lasker. “Everything couldn’t just all be going the same direction, so the brushstrokes had to live in 3D space. Once you move that camera it has to still work and can’t look flat, and then you needed the ability to put brushstrokes wherever you wanted, to really make it effective.”
PatchyBomby: brush projection
A brush projection tool Imageworks built for the film was called PatchyBomby, again inside Nuke. The first film had something similar–PatchyFindy–which was used to ‘mosaic’ up an image. The new tool overcame a limitation of the original by being able to blend between layers or rendered layers to represent varying stroke opacity or bleeding colors.
Curve System: fine control
Curve System is another Imageworks Nuke tool that allowed artists to create hand-drawn strokes. It was used for very specific and deliberate detailed strokes that could be applied to animated imagery, whereas Stoke System [discussed above] was generally relied upon to distribute tens of thousands of strokes in a scene.
Fyber to the node
Imageworks’ proprietary hair tool Fyber formed a new workflow through Katana to Arnold specifically to groom brush strokes onto the character Miguel, who has very specific contours. A script inside Katana sampled color and other data to attributes that were passed to Arnold (Imageworks’ version of Arnold), where a shader worked out final brush shape, color and visibility.
MaskToInk and PigmentMerge: painterly spots and pigments
A tool called MaskToInk allowed Imageworks artists to render a simple mask as if it were made of wet ink. The tool then had mappable controls for where and how much ink should spread based on how wet or dry the surface is. Meanwhile, PigmentMerge is a tool that Imageworks used to composite via subtractive color mixing as if they were mixed pigments.
Rebelle: animating paintings
An existing digital painting tool called Rebelle made by Escape Motions was adapted by Imageworks within Houdini to help deliver non-photorealistic looks that simulated the movement of brushes and flow of paint. This was particularly useful in simulating watercolor and oil and acrylic looks, which were generated with Rebelle and then animated over time through Houdini.
Flixiverse: story into 3D
While the tools above were created by Imageworks, Flixiverse–an app that allowed story artists to access more complicated 3D models–came from the Sony Pictures Animation (SPA) tech team. It allowed artists (including even the directors) to rapidly place 3D models into their storyboards for a closer approximation of the final scene.
Oto: dealing with dialogue
Another SPA app was Oto, which allowed hours of recorded dialogue to be automatically transcribed with AI techniques, and deliver the audio files to individual lines. This reduced transcription times for assistant editors.
CASE STUDY #1: Making Mumbhattan
At one point in the film, the heroes find themselves in the home dimension of Pavitr Prabhakar known as Mumbattan (Earth-50101). A catastrophic building collapse occurs through deep layers of the vertical city, wreaking a massive amount of destruction. “It was a huge effects sequence,” notes Lasker. “You have smoke, inside the smoke, you’ve got dark matter, you’ve got debris, everything.”
Imageworks built scores of skyscrapers for the sequence and also broke the city down into subsections. The scenes made use of tight shots of buildings and many wide views of the city. “We had to use our Sprout instancing system to help build a lot of that,” describes Lasker. “And we had matte painting to build back from 2.5D work, although for the most part we did render everything as 3D.”
Lighting the expansive environment was particularly tricky, adds Lasker. “We always want to read the character’s performance, be able to follow the characters, and these are fast shots. We’d have to light them and layer them. We would always split them out as the far canyon and the close canyon, and go into different color families between the weather, the distances. You’d either be in the cools and the blues and the greens or the warms and the reds and the yellows, and then always play the characters dark over light or light over dark.”
“I worked closely with the lighting teams to come up with rule sets to light these shots because they were so massive and so busy. So much of Mumbattan is about density with wires and vehicles. so you had to use the complexity against itself and simplify it wherever you could.”
CASE STUDY #2: Vulture attacks
On Earth-65, Gwen confronts a version of the Vulture from an Italian Renaissance-themed alternate universe with the aid of Miguel O’Hara and Jessica Drew. While Vulture ‘appears’ to be almost made of sketches and parchment paper, he was of course a fully 3D character.
“He’s actually one of the most complex characters we’ve ever built,” remarks Lasker. “We started these series of meetings at work called ‘character discussion meetings’ because of him, and then it ran into other characters. We had so many characters that required every aspect of the pipeline to be involved. But he was the first one of them and so complicated.”
Vulture was built completely in 3D and rigged like a normal CG character, with Lasker noting that “we could even use him in a live-action Spider-Man movie if we wanted to. I mean, he’s really there with pulleys on his arms that link up to all the pulleys on the wings. There’s all this wooden framework and parchment built between them, and he has his physical wings, he’s got flowing hair, a mask, beak, fully emotive mouth, feathers. He has a mane of feathers behind him, and then his whole chest and torso is covered in, and they all simulate and they all move.”
What makes him more of a ‘Spider-Verse’ character, of course, was the stylized use of parchment and sketch-like qualities. This was enabled with around four layers of line work on top of his surfaces. “We’d have our typical texture painted detail in there that we could modulate and remove if we wanted to, explains Lasker. “We’ve got paper texture that came from a tool called Dual Rest, a way that we project textures on and enable dissolving between textures so it never looks like it’s locked on or that the texture is just painted on.”
Then, Imageworks’ new Kismet line tool inside Houdini came into play, for inklines and outlines. “So,” summarizes Lasker, “you’ve got textured lines, you’ve got layers of different Kismet lines, and you’ve got hand-made lines all forming part of Vulture. We would even have lines outside of his body that would animate and move and redraw, and use lines inside. It was just layers and layers of ‘stuff’ that added up to this parchment.”
Imageworks even experimented with a kind of ‘Nickelodeon’ effect over Vulture to make him appear somewhat like a flip book. “You can actually see it when Miguel shows up for the first time,” states Lasker. “You’ve got that closeup of Gwen, that slow-mo close-up, and you got that close-up of Miguel going by, and you see the Vulture’s wing flickering like a flip book. We put a little bit of a light behind it, like old cranking Nickelodeons. It looked like he was redrawing every frame. In the end, every shot of him was not quite the same. That’s the fun of it.”
