How tennis doubles, face replacements, machine learning, CG courts, balls and rackets, and even plexiglass helped make the stylized gameplay of the film possible. From befores & afters print magazine.
Luca Guadagnino’s Challengers features significant tennis play by its principal characters: Tashi Duncan (Zendaya), Patrick Zweig (Josh O’Connor) and Art Donaldson (Mike Faist).
While the actors playing those characters undertook tennis training and performed many of the playing scenes in the film, Challengers also took advantage of several visual effects techniques to bring the needed authenticity to the story.
“Keeping the actors authentic to tennis was very important,” says visual effects supervisor Brian Drewes, whose Zero VFX handled the majority of the film’s visual effects work. “We didn’t want them to look like bad tennis players. We wanted them to look like great college level-going-onto-professional level tennis players.
“The actors worked extraordinarily hard on things, as far as their training and the routines that they would follow. But we knew from the beginning that we really wanted to be in close with them. We wanted the camera to be really kinetic. In fact, we were shooting tennis in a different style than anything has ever been shot, I think, including that final crazy scene where we take the point of view of the ball.”
Anyone for face replacements?
We’ll come to that stunning ball POV finale of Challengers, below, but first, Drewes explains more about achieving the authentic look of professional tennis rallies. “To do that work, we had tennis doubles on set, and then we did face replacements on the actors using a traditional 3D pipeline along with machine learning to help with some of the close-ups.”
“During filming, the stunt doubles would go first wearing dots on their faces, since that was always really good reference for the tennis action,” continues Drewes. “It also helped the camera team map out exactly how they were going to work out that specific beat. Everything was pre-planned, down to the tee, with storyboards. Every tennis moment was scripted and storyboarded. Then we would have the actors walk on and do the best that they could. They did a great job of matching to the doubles.”
Here, the actors would not actually be hitting tennis balls, unless they were picking a ball up. Instead, Zero VFX crafted most tennis rallies with CG tennis balls. “We really wanted the actors to focus on the style of their strokes, not worrying about whether the shot would go in,” says Drewes.
To ready the face replacement work, the actors went through a photogrammetry scanning process. This was completed by The Scan Truck. “We went through the typical face body scanning, and then modeling via our traditional pipeline,” outlines Drewes. “Then at the end of that pipeline, we had a lit and rendered CG face. That’s when we did some AI and ML processing on that rendered face, and would then hand that to a compositor to be able to blend those two passes together.”
The face replacement work then had a number of challenges. One was that in some scenes the character of Patrick has a beard. Another was that the tennis doubles did not necessarily closely match the look of the actors. “It meant that we had to deal with times when, say, one tennis double’s head was much rounder,” notes Drewes. “For that, we’d end up doing full head replacements at times rather than just replacing key features.”
“We also keyframe animated all the faces, once we had those in place, and we had some occlusions to deal with as well where tennis rackets are swiping over faces. That’s when we applied the machine learning process on top of the 3D in order to really get that next few per cent that really locked in all the facial features that you would expect to see from the actor.”
Taking the gameplay further
Zero VFX was also able to utilize digital visual effects to take some unique shots further, as Drewes explains. “Luca really wanted to be close to the action and to give the audience some moments of ‘What just happened?’ So we had a bunch of shots where the camera would’ve been so close to our actor’s swings that they would’ve nailed it with the tennis racket. We ended up shaving off and just giving them the stump of the racket to use for these shots and then we tracked the CG racket onto it. I think it just really pulls you in as the viewer when you’re seeing those shots. It doesn’t feel like a visual effect. It just feels like, ‘Oh, that was a cool way to film it.’ Whereas, in fact, those moments generally were enhanced by visual effects.”
Another enhancement to the viewing of tennis gameplay included ‘under the court’ shots of the players, as if a camera is looking up to them from underneath. “That was done plexiglass,” states Drewes. “They actually did that as a secondary shoot in Italy. There were some face replacements on those, too, and we did some clean-up on that plexiglass since it became quite scuffed over time. Plus we ended up adding the tape line marks to the shots.”
A further visual effects effort revolved around environments, especially for the centerpiece match at New Rochelle. For this match between Art and Patrick, production filmed on location for three weeks. The match is intended to occur in the early evening between 5pm and 8pm, but, of course, over the three weeks, the filming took place at different times of day and in different lighting conditions. This required a range of VFX augmentation.
“The first AD did a really great job keeping track of all the changes in conditions, but nature didn’t always play nice, and there was no way to put a tent over the whole court,” says Drewes. “There’s just no way to do that practically. So, we ended up doing a very, very substantial photogrammetry scan and LiDAR scan of that court and relighting a large number of those shots in order to keep them continuous.”
“In fact,” continues Drewes, “a good 50 per cent of those shots saw us relight the court, adding shadows of trees where they’re supposed to be, and keeping that continuous throughout the cut, changing general lighting directions on certain objects, replacing things in CG if they were blown out.”
In one shot, the camera comes in from behind the umpire, past his left shoulder and then across the tennis court net as if on a crane arm before settling on Tashi’s eyes watching a game. This move was a stitch between three different camera moves.
“We previs’d that,” advises Drewes, “because there was no hiding it whatsoever. There’s nothing to wipe the frame with. So the previs was what was followed on set. Also, we realized that we would have people in the audience looking back and forth at the tennis action as the camera was going across, which was another constraint. We had to work out, how do you seam together that action in a continuous piece and find a cut point? It doesn’t look like much when you watch it, but it was a total rebuild.”
How to be a tennis ball
A rolling tumbling tennis ball–and matching camera–is served up as the film’s near-finale sequence, following the high-pressure moment between Art and Patrick from the POV of the ball. To get started on this complex scene, which would involve live-action filming and several CG elements, Zero VFX began with previs, and curiously, the ‘sound’ of a tennis match.
“We took the audio bed of a tennis match as our measure of the speed of a tennis rally,” begins Drewes. “The ‘speed’ that you hear for when those returns are happening, this was in our previs. The idea was, whenever we were debating, ‘Is this too fast? How dizzying is this going to be?’, we always had the ground truth from a real match. Then we could choreograph the cool camera moves to match, but as long as the audience was understanding what they were seeing spatially, then it worked.”
Of course, this is a moment of heightened reality. A spinning ball, notes Drewes, spins at about 10,000 rpms, but that was never going to be part of the shots since the audience would not make out any of the action at that speed. Still, the style of the scene allowed for some dizzying moments, all of which began with a base of live-action.
To film the live-action, Drewes second unit directed the scene with the tennis actors and tennis doubles. Here, he shot digitally (most of the movie was shot on film) to allow for higher resolution and more manipulation in post, especially for slow motion parts. Zero VFX had also produced techvis out of the previs to aid in how key moments could be filmed, how the cork-screwing camera would move, how high the camera should be off the ground, et cetera.
“I had one day on a Sunday to film this, and it was quite a sunny Sunday. The DP gave me a window between 3pm and 8pm to shoot it. The only equipment we ended up having for this was an ARRI Alexa LF on a 30-foot Technodolly, not long enough to get down the court. So, we ended up breaking it up and block-shooting it. From one side, we shot all the scripted action. The first segment, from one direction, we’d shoot, say, volley 1, 3, 5, and 7, and the start and end points. Then we marked that in 3D space and moved the crane base to shoot the second leg of volley 1, 3, 5, and 7. We basically do-si-do-ed our way down, and it was two or three shots per full volley down. It was three different stitches for each full length travel of the court.”
“Then,” adds Drewes, “we did that back in the other direction, and then that was our day. Luckily on set we had our VFX editor who was able to help work out all the seam points. We wanted to always make sure that we had enough run-in to the seam, so that we were in the right spot, and that we had enough velocity on the crane arm to be able to keep generally a continuous speed through those seam points.”
“For the actors, we ended up having them work as best they could at slower speeds,” comments Drewes. “We’d say, ‘Okay, generally, move half your speed.’ And we would immediately look at the result with our VFX editor and say, ‘Okay, that doesn’t look right. Go faster, go slower.’ We also shot pick-ups of our lead actors over bluescreen, so that for the moments where we’re seeing our actors from further away, we could take a 2D element and scale into that for the first part of the scene.”
In the end, 23 individual shots formed the final 24 second ball POV moment. The sequence would make the most of Zero VFX’s use of the digital court, face replacements and CG background crowd that came from scanning extras. “A lot of the moments,” says Drewes, “became an amalgam between our CG environment and the practical camera shoot. There’s a lot of moments that are all CG and a lot of moments that are all practical.”
“I think that scene and many of the others in the film were able to show tennis, and the shooting of tennis, in a different style than anything has been shot before,” Drewes reflects. “I haven’t seen too many sports movies that shoot as close and as intimately as we did, because it’s really hard to do! The ball’s going 120 miles an hour, you have to keep up with it. Doing it on film, too, we knew that there was always going to be this challenge of how do we get the camera to keep up with the actors? How do we keep them looking the best that they can? That their strokes are looking unforced and professional looking? There’s so many techniques that we ended up developing in order to keep that be the case.”
