Luma Pictures delivered a digital version of the Candyman figure for the film.
During filming of Nia DaCosta’s Candyman, an actor played the titular figure in special make-up effects, performing scenes where he would be initially only glimpsed in reflections, and then carrying out some brutal kills.
While some digital augmentation had been planned for the character, it was ultimately determined that a fully CG version of the Candyman–Sherman Fields–would be necessary to truly depict the brutal manner in which he had been killed years earlier by racist police officers.
Luma Pictures, one of the VFX vendors contributing to the film, working with production visual effects supervisor James McQuaide, took on several CG Candyman sequences. Here, visual effects supervisor Andrew Zink, who is one of the nominees for Outstanding Supporting Visual Effects in a Photoreal Feature in Candyman (alongside McQuaide, Josh Simmonds, Drew Dir and Ryan Evans) tells befores & afters about Luma’s challenges in crafting this new synthetic performance.
b&a: I was surprised to learn of the extent of visual effects in this film, it doesn’t feel like a VFX film at all.
Andrew Zink: Yeah, maybe that’s a good starting point, is that not a lot of people will give the credit that a horror film will typically deserve in the fact that a lot of them don’t get the budgeting that a normal large blockbuster would get. So, you can take that same assumption with Candyman, and in this case, with Sherman, or Candyman, being the case.
When they went to originally do the principal photography, the intention wasn’t quite yet to have his face be fully digital or even him necessarily be fully digital. So, they had an actor to play him. He wore a prosthetic mask, and the intention was to just augment that prosthetic mask in two-and-a-half-D to try and patch some of the eye holes for the actor, or maybe enhance some of the blood on it to make it feel like it might be more wet, or change in some of the texturing, the spattering, and really not go much more than that.
But, over time, as the filmmakers were pushing production along, and as they were editing it together, they didn’t feel like the prosthetic justified the beating of Sherman well enough. It just didn’t come off as realistic or visceral or violent enough. Also, the thing with Sherman is that you want to always capture that moment frozen in time, as if you were looking at a very brutal crime scene photo. The intention is to be scary and provoking, but there’s subtleties in that they’re trying to tell this story of this man who is unjustly killed by the police, and that is the result of the prejudices of his killing.
b&a: What I think is interesting is that, because it’s not something that was necessarily planned from the beginning, I mean, if you were doing a digital human, you would, these days, get scans and approach it in a very particular way. But because you were doing it a bit differently, did that mean that your approach was much more brute force?
Andrew Zink: We did have scans of Sherman, and that was great to start off with. So we were able to mock up and recreate his body and his wardrobe verbatim, as if it were in the plate. But the head and the face was a whole different ball game altogether.
What we did was start to take a dive into looking into crime scene photography. Now, in working with artists, you need to make sure you’re caveating some things like, ‘We’re about to have to look at some pretty graphic material. Apologies.’ Some of this stuff is pretty disturbing, so making sure that your team is first and foremost okay with reviewing that kind of stuff is important. You have to study it. You have to understand, ‘Well, how does the face deform if it’s had blunt force trauma?’
You end up having to study and dig up and find pretty graphic content, but again, we didn’t want to do a disservice to it because it needed to be graphic. It needed to feel as though we were conveying that point that the storytellers were trying to convey.
We went to work with basically taking crime scene photography, blunt force trauma reference, car crash reference of people who have unfortunately had some pretty serious injuries, and trying to take elements of those to recreate Sherman’s face, starting off with the actor and then starting to model in and deform parts of his face so that it looks like he’d been clobbered.
And then it got down to the point where we said, ‘Alright. Well, we know that we’re going to go full digital on this. We need to just continue on with what the nuances and the subtleties that would go along with having a hero digital double asset.’ So, everything down to the modelling of the deformities from his face being beaten to the peach fuzz on his face, his hair, his eyelashes, eyebrows, all of that was all done with our hair system, as well.
And then, just trying to recreate the bruising and the post-traumatic injuries that skin retains from having gone through that trauma is something that we were trying to adequately put into that asset to make sure that you could read that. So, long story short, we basically had to study some pretty brutal stuff and try and recreate that as best we could.
b&a: How did you approach facial animation here, given I’m guessing you didn’t have any kind of on-set facial capture?
Andrew Zink: Originally, the face wasn’t intended to be animated very much. It was just about augmenting the plate face, and then as the project snowballed, and we started working shots, we’re like, ‘Yeah, we’re going to need to have some facial performance,’ especially as there’s some pretty significant closeups of Sherman. Normally, you’ll get head-cam footage, and you’ll get tracking marker information, all sorts of different kinds of reference photography to be able to rotoscope or rotomate anything that might be happening in the nuances of the performance. Of course, none of that was available.
So, Raphael Pimentel, our animation supervisor and his team–it was up to them to be able to bring the subtleties and the nuances of Sherman’s face and emotions to life. Granted, he’s not delivering dialogue, and he’s not going into any kind of heavy emoting. But still, there’s subtleties that needed to be built into the rig and be able to be sculpted so we could get an interesting smile, an interesting squint, or any kind of other subtle emotions that we wanted him to be able to portray, which again, leads into him being a little bit more disturbed that we just had to film.
Raph would film himself and try and capture what he felt like the performer would be doing, and then he’d distribute that to his team. It’s not too dissimilar to what you would do with an animated feature when you record your performance and then try and recreate that.
b&a: Just to be clear, is it a completely CG digi-double, or were you relying on any projections from the plate or anything like that?
Andrew Zink: Totally CG digi-double. From the top of his head, the top of his hair, down to the bottom of his shoes, fully digital.
In the hallway scene where Anthony and Sherman are interacting, and Anthony sees Sherman, originally, it was shot with a performance artist. They choreographed this whole scene where he’s trying to get the reflection to do the same thing that he’s doing. And it worked great, and we had that as a reference, but we were basically going to just replace it in full because, at the end of the day, we ended up enhancing Sherman’s wardrobe with more additional blood than what was in the practical one.
We were able to use the practical scan and reference photography of that to be able to just start as our basis. And then, when it came to dress him up with more of the brutal elements, that was the artistic side where we were trying to move forward with something that felt a little bit more scary or a little bit more graphic.
With reference photography of Sherman doing his performance, we could rotomate that as a starting point. But then we wanted to be able to make sure that it was an exact replica or an exact copy of Anthony’s performance, so we would end up just studying what he was doing and then try and get the timing and everything to match.
b&a: What were the main tools you used for this work?
Andrew Zink: All of the modeling and animation rigging was done within Maya. In terms of our lookdev and lighting, a couple years ago we migrated over to Katana. Our facility is now a full-blown Katana / Arnold rendering pipeline. All of Candyman’s work was done through that pipeline, through rendering, using our USD pipeline.
In terms of simulations, Houdini is our Swiss Army knife for all of those. There are some pretty brutal blood sims that we did that were all done within Houdini. Then we’re doing cloth sims within Maya, and comp’ing everything within Nuke.
b&a: Tell me about the mirror scenes Luma Pictures worked on, in terms of planning and executing them.
Andrew Zink: It’s interesting, I think everybody has an idea of what they think it should be doing when you’re looking in a mirror, and until you actually start to break it down and physically mock it up, sometimes you have to build a little bit of a sandbox test to show people.
So, if we have someone who can see Sherman in a mirror, we can do a couple different things. You can try and put Sherman in the room and, if he’s supposed to be reflected in the mirror, you can try and physically accurately render that. And by that I mean putting him where he would be accurately within the room, i.e., have a mirror really reflect that CG asset, then render his reflection.
You can do that. But…there’s no real point in doing that for a lot of the shots that we were doing. Because outside of the novelty of being, ‘Hey, we actually rendered a real reflection,’ you have a whole bunch of issues with it. It’s hard to be able to understand what it’s doing before you render it. You can’t really get direct feedback to the artist or the animation teams on how it looks without being able to understand what it’s reflecting. And we always want to be able to have a pretty tight feedback loop and high iterative process to be able to move through the work in a meaningful way.
So, in this case, we ended up building a mirror rig of Sherman. The idea is, what would be his right hand with the hook now becomes his left hand with the hook. For all intents and purposes, you basically flip Sherman to be a mirrored version of himself. Raph and his team would use the mirrored rig of Sherman and place him beyond the mirror, as if we were building it within the space here. We would be pushing Sherman into the mirror’s space, so to speak, and getting what felt like a correct perspective of Sherman.
If there was any conjecture like, ‘I don’t know. Something’s not quite right. It doesn’t feel like he’s the proper depth for what he needs to be in,’ then it goes back to the sandbox anecdote. We’d say, ‘Can we do a real reflection of him? Just put him in there, reflect him, see where he shows up at, use that as reference for our team to get the proper depth for putting him beyond the mirror.’
That’s how we would start to try and get around this. That way, our anim team was able to have direct feedback. They could see him within Maya’s viewport, animate him, do everything that they needed to do. Except we were just using a flopped version of Sherman.
b&a: Right. It’s like a puzzle you have to solve in your head in planning, on set and also in post.
Andrew Zink: Absolutely. It’s a really strange.
b&a: To sell the effect, to help editorial, to make sure it’s scary, were you essentially providing any kind of rough comps or animatics, or was that already worked out already for those mirror shots?
Andrew Zink: Yes, we were doing animatics and rough comps. In the hallway scene, they obviously had footage they had shot, so that wasn’t necessarily an issue, but in scenes where that wasn’t prep’d that way, it was filmed with real mirrors. We needed to be able to get something in front of the client as soon as possible because we wanted to get performance notes. We wanted to be able to get feedback. So, it was imperative that we were able to put together playblasts or anim vis, basically, to show them, ‘Hey, this is what we’re thinking.’
Then it was just a back and forth on, ‘What would Sherman be doing? What would his performance be doing to make him feel more menacing, more haunting?’ And that was another thing that we needed to be able to try and follow through with, was that his locomotion was meant to be disturbing. So, we came up with this idea of whenever you would see Sherman moving, if you happen to see his feet, you would generally see him zombie-like, floating and dragging his feet along. And his feet would be catching just ever so slightly some of the ground and tapping along.
He is a ghost at the end of the day. So, if he were just to be walking, where’s the supernatural kind of entity within that? We wanted to take it a little bit step further. It kind of conveys a limp body. That was another story point that we were trying to pull and convey into the performance.
b&a: Obviously, dangling and dragging you can do with the CG render of this character. Did they try and do that live action, also?
Andrew Zink: No, not that I saw in any of the footage that we had. A lot of the times, he would be filmed, and he’s not very animated. He’s quite still in his performance until he’s actually going in for a kill, and then he’s quite brutal and powerful. So, nothing was ever mocked up practically for that. That was definitely all just done by riffing on ideas of, ‘What can we do to make him feel supernatural and scary, but also what would feel realistic as well?’
