Sony Pictures Imageworks on the many characters it had to craft, including three Spider-Men–for the final ‘No Way Home’ showdown.
The third act final battle in Jon Watts’ Spider-Man: No Way Home includes a large number of ‘multi-verse’ characters–there’s three Spider-Man, Green Goblin, Lizard, Electro, Sandman and Doc Ock. Bringing them together in a massive battle set amongst the scaffolding of an under-renovation Statue of Liberty was Imageworks, led by visual effects supervisor Chris Waegner.
Here, in an in-depth befores & afters interview, Waegner examines each of the different character requirements for the sequence, and the vast environment build.
Spider-Men: new and old
b&a: Starting with the different Spider-Men, I’m curious about the challenges you had there just in terms of scans of the actors, but also building characters that obviously Imageworks had built previously?
Chris Waegner: Firstly, It was a very exciting opportunity to work on this film having worked on quite a few of the previous Spider-Man films while being at Imageworks. Having worked with each one of the actors and each one of the directors, each of them brought their own style to the franchise and to the big screen for the fans. It’s such a passionate fan base which spans across many different age / gender groups. There will be children seeing this film with Tom Holland as Spider-Man for the first time, while there are folks who are older, maybe comic book fans, who originally saw Tobey Maguire as Spider-Man on the big screen, or years later saw another instalment with Andrew Garfield in that role. It was very important to us to do the franchise justice. I know Director Jon Watts and VFX Supervisor Kelly Port were equally as passionate about this franchise.
We began re-building our digital Spider-Men assets, utilising detailed LiDAR scans and photography. The Sony Imageworks team has built these characters in the past for previous films, but since technology has evolved, most of these previous assets were used for proportional reference and it became clear, we had to start over and rebuild new Spider-Men. Another key decision in building the new Spider-Men was that the actor’s physical appearance had changed as they aged. Luckily for us, the studio has maintained it’s in-depth photographic history of what the original suits look like for both Tobey and Andrew for our build process.
Tom Holland’s new suit on the other hand had its own set of challenges. During principle photography, filmmakers decided that Tom’s character was going to have what would be called the “Hybrid Spider-Man Suit”. So for our sequences, we replaced the practical suit Tom was wearing in camera with a newly designed digital suit. His new digital suit is composed of fabric with metallic inlays and overlays all woven together. Replacing Tom’s suit in every shot was a very laborious process which involved various in-house techniques across many departments in order to ensure the final look would hold up to intense scrutiny. It was critical to our filmmakers that this new digital Hybrid Spider-Man suit accurately mimic Tom Holland’s subtle underlying muscle movements, twitches and physical interactions with onset actors.
There are so many subtle things Tom does when he’s physically interacting with Ned and MJ. All the suit wrinkles and physical compressions when he hugs someone, these were captured in camera, and now had to be duplicated in this new digital suit. A relatively straightforward blue screen shot now took on an additional level of complexity because of the suit replacement. We’d start the process by matching to the set lighting, then proceed with highly accurate rotomation to match all the physical interactions he might have with MJ, Ned or Doctor Strange, or his environment. Things you take for granted, like, suit wrinkles or compression of his feet all needed to be matched to the plate photography. When filming an actor all these subtle details are captured in camera, but when the actors body is replaced with a digital body, all the subtle nuances need to be duplicated and put back into the shot.
b&a: I also thought you did an incredible job in helping to tell the differences between Peter one, Peter two, Peter three. What conversations did you have with Kelly and also the director about different behaviors or swinging or other things that would help tell the difference?
Chris Waegner: Being able to identify each individual Spider-Man was very important to both Jon and Kelly. Early on it was quite challenging because their suits are pretty similar in tonal values and since our sequence takes place at night this only exacerbated their individual readability. It became pretty obvious once we started lighting shots what we had to do. We focused our efforts on really bringing forward each of the Spider-Mens physical personalities or their individual spider-style, if you will – how they swung, how they shoot webs, their physical proportions when they achieve an iconic pose. So our animation supervisor, Rich Smith, who has also worked on several Spider-Man films, had our animation team review the previous films. We studied clips of each actor and reviewed their individual spider-style from these films. We then layered in these signature performance characteristics so that each Spider-Man had their own unique style when seen next to each other.
b&a: Was motion capture part of any of that Spider-Man animation?
Chris Waegner: There was a motion capture shoot and the animation team used those performances when possible, but as with most capture shoots, it’s a great reference source assuming everything is choreographed correctly. The data was a great tool to have at our disposal should we need to layer it into a shot. What tends to happen is a bunch of motion capture data is collected, whether it’s Tom or his stunt double. We incorporate this data into our initial shot blocking. As the shot evolves and changes we try to utilise as much of the captured performance as possible. Inevitably, our animation team determines how much of the initial capture data can be woven into the shot based upon creative direction from Jon and Kelly. The motion capture data can help greatly depending upon specific shot needs. In the end, it’s a tool we utilise when needed. In our case, the animation team was very focused on incorporating each of our heros’ physical performance characteristics and that’s where the team’s attention to individual Spidey-details really shined.
b&a: You mentioned scanning the actors and the kinds of builds that you had to do, but to what level did you need to build photo real digi-doubles of Tom, Tobey and Andrew? How far did you need to take it?
Chris Waegner: Well, I think when you work on a show like this, it’s not uncommon for the client to ask for something that makes a shot incredibly spectacular. So, being able to rely on these digi-doubles to a high performance level allows us to help the filmmakers achieve their vision. Our Modeling and Rigging teams are very influential in the initial build of the characters. They work closely with the Animation team in creating the digi doubles, ensuring they will meet the filmmakers performance needs. When you look at a hero like Spider-Man, there are these classic comic book poses, or signature poses from past movies – these have become iconic in history for this character. The reality is, most normal folks can’t possibly get into some of those poses unless they’re an extreme contortionist.
So, maintaining physical proportions, and accurate muscle volumes for the character is extremely important. He needs to do physically impossible things and not look broken – whether while climbing up a wall or getting into a crazy web shooting pose as he’s flying through the air. It starts at the beginning – these digi-doubles really need to have a stable foundation built with input from many departments all working together while focused on the same goal. The physical demands our animation team put on these spider-men digi-doubles is huge, and it gives our filmmakers confidence we can meet their creative vision.
A new-look Lizard
b&a: How did you approach The Lizard in the sequence?
Chris Waegner: The Lizard (Dr. Curtis Connors) from “The Amazing Spider-Man” movie and had a very specific design to him unlike some comic book adaptations. For this movie, we started off exploring some new physical changes but in the end our filmmakers kept him quite similar to the previous design, incorporating just a few subtle changes. It was important to our filmmakers that we explore different facets of The Lizard personality in this film. It was a lot of fun for the team to “unleash” The Lizard at Liberty Island. He was kind of like the proverbial “bull in a china shop” – full of reptilian rage while running through the construction area, tearing through the scaffolding and chasing Ned and MJ. In contrast, earlier in the film when he’s in the Sanctum, he talks with the other villains, inquires about his fate and shows a lot of personality with subtle facial features and his body language showcasing a different side of The Lizard.
b&a: Was he effectively completely keyframed or was there some sort of motion capture that could be used as a basis?
Chris Waegner: The Lizard was essentially all keyframe animation. There was no motion capture on him for this film, partially because of his physical size. When they built the scaffolding on set, it was built to real scaffolding proportions for our actors to perform upon. The Lizard is huge, essentially eight feet tall. We tried to contrast our hero’s performance to that of The Lizard during the scaffolding chase sequences. So, while the Lizard became a physical wrecking ball charging through scaffolding, our spider-men utilised their unique skills throughout the environment while evading him – climbing on the ceiling, leaping from wall to wall or swinging off the structure. This juxtaposition in character performances was a lot of fun for the team. His physical size allowed for this fun destruction – throwing boards, beams falling down, poles bending, every time he takes a step the boards flex due to his sheer weight. These tend to be real subtle nuances within a high speed action shot but it’s that last 5% that can bring a shot to life.
b&a: Was there anything being done in particular for facial animation there, especially any kind of reference for the actor Rhys Ifans?
Chris Waegner: Well, because Imageworks had worked previously on “The Amazing Spider-Man” with The Lizard, we had an extensive inventory of reference photography which had been done for the original movie. This reference was restored and basically became a starting point for our in depth character study. We used the original Rhys Ifans data as reference, but since there were some proportional changes in the new Lizard design, it could only be used as reference. With so many villains, his screen time was limited and since his psychological component had been already established in the previous film, this allowed us to focus more on his physical performance. During the final battle he became solely focused on trying to stop our heroes and retrieve the relic.
Revisiting Sandman
b&a: And what about Sandman. Kelly Port had a fun story about how there was some collective PTSD at Imageworks from previous sand animation done for the third film.
Chris Waegner: Well, yeah, I would tend to agree with what Kelly said in that regard. When we did the original Sandman in “Spider-Man 3”, Imageworks was pushing the envelope with the computer technology and software at that time. As luck would have it, computer technology and simulation software has gotten much better with time. Having the ability to simulate hundreds of millions of grains of sand in a shot, would’ve taken days and days to calculate previously. With new technology, we were able to optimise these simulations to our advantage, most notably our ability to quickly iterate on multiple versions of the large Sandman character for our sequence. Once a character look had been established, we were able to quickly iterate on these complex Sandman FX simulations within shot context.
For some of the scenes with our large version of Sandman – there are moments when he’s more of a physical storm, made up of not just sand but earthly elements all mixed together being drawn from the island environment. This large size allowed us to explore a chaotic aspect to him – as material is drawn from his surroundings to help structurally form him, it is also flying off him much like a tornado of sorts, or just this giant storm of a villain. That concept became a big undertaking, since it was important for us to visually convey the chaos of Sandman when presenting to Jon and Kelly. Fortunately, our early FX optimization development allowed us quick simulation iterations thereby securing an initial concept buyoff of him being this physical storm at a larger scale.
During shot production this became a bit of a learning process, since it wasn’t possible to creatively submit animation with a giant Sandman FX simulation for creative buyoff. Based upon our early character development of Sandman and his storm, a trust had been established with the filmmakers. This trust allowed us to present proxy versions of Sandman within an animation context, once approved we could quickly move onto the gigantic FX simulations. Even though it might take a couple days for a shot to run though the entire pipeline (animation, simulation, lighting, comp) the end result was film quality in a fraction of the time. I believe this approach was quite successful in allowing our filmmakers to see high quality shot work quickly.
b&a: And again, is it because of the previous history of working on the films that you were able to tackle that facial animation or was there any kind of capture from Thomas Haden Church at all?
Chris Waegner: I believe our previous work with this character allowed us to explore and navigate in the most efficient way possible. We knew from experience what to avoid and what to pursue with new technology. As far as the facial reference goes, again, it was fortunate for us having worked on the previous film, we were able to restore some reference, and once online, that became our starting point. Yet, we were reviewing visual material that was about 14 years old and so much has changed since then. For large scale characters like this, there’s a fine line between looking scary vs. looking cartoony, it was a constant balancing act that we did. When you have a giant character made out of a sand storm, you need to be able to read his facial expressions, but not exaggerate them so much that it takes the audience out of the moment. In the end, it really came down to our talented team of artists who strived to get in all the details needed to make Sandman.
Electrifying
b&a: I really liked Electro in these scaffolding scenes as well. There was something really nice about these lightning bolts. They felt very handcrafted.
Chris Waegner: The updated electrical look was done by our FX team in Houdini, and then a treatment was applied by the compositing team. The previous incarnation of Electro’s electrical effects from “The Amazing Spider-Man 2” had a different style altogether and for this show we pitched the electrical effects being more violent and staccato with moments of pure energy. The idea being, that when he zaps from location to location or he’s hovering taunting our heroes, the electrical effects are almost disorienting for our heroes. There’s always an electrical storm in the air surrounding Electro, almost like being near a giant Tesla coil. It opened up so many creative possibilities for the artists when lighting these shots, even though he may not have been in frame, once the electrical look had been established, we had the ability to do dynamic lighting which posed the threat of him nearby.
We did a deep dive researching Electro’s updated look, reviewing many different real electrical events while trying to determine what was best for him. We even went back to comic book reference and incorporated some of that into his new look. If you just look at lightning references, you can find all kinds of crazy visual stuff, there are so many different types of lightning. Electricity presents itself in so many different forms, we ended up creating an updated stylized look we felt was both threatening and beautiful, while offering us some cinematic freedom in a night sky for our sequence.
That trailer
b&a: There was a really fun thing about the battle sequence, wasn’t there, where a trailer didn’t give away the multiple Spider-Men in one of the fight scenes, as Lizard got hit in the face. Is there anything you can say about that in terms of delivering a shot without one of the characters? I just thought that was such a fun thing that the internet went wild about that.
Chris Waegner: As filmmakers we are often asked to deliver WIPs and other various media to the marketing teams. We pretty much give them what they’re requesting but we don’t see the final assembly until it’s released mainstream. I know the internet went crazy and there were many rumours about it.
The set piece and environment
b&a: While we’ve concentrated on the characters here, there’s also a huge set piece in terms of the Statue of Liberty and then there’s the New York environment. I just wanted to make sure I give a shout out to the Environments team for how that came together.
Chris Waegner: The final battle itself takes place around Liberty island and it was a gigantic model to build. They built portions of the scaffolding on set, which they then shot practically. These physically built set pieces became our material and style guide as we digitally built the scaffolding around Lady Liberty. In the movie, the Lady Liberty location is basically under renovation. Fortunately for us, this type of work has happened in the past and there’s a tremendous amount of historical photography reference which we used, as well as, very accurate LiDAR reference. Once we had our reference in order, our Modeling team began the Liberty Island build. It was important that we adhere to structural details, since we knew this would play a crucial role in our destruction simulations. The physical interaction between various building materials (ie. wood, stone, steel, alum, etc) was vital in showcasing the much needed visual complexity for the destruction in our sequence.
The island build would also encapsulate everything at ground level, there’s a giant construction area, shore line, all the island building structures, watercraft and then there’s the city out in the far distance. We utilised some photographic reference of the city when needed, but most of the shots showcase our digital environment. Everything was built at a high level out of necessity because at the time, we weren’t entirely sure where most of the action would take place. It was a tremendous undertaking for the Modeling and Environments team.
