MR. X on the combination of practical and digital for the creature Larry.
In director Jacob Chase’s Come Play, a monster called Larry enters our world through electronic equipment, ultimately terrorizing the young Oliver (Azhy Robertson) and his family.
As it has done on several recent films, MR. X was tasked with complementing an on-set puppeteered practical creature with CG facial animation and occasionally a fully digital performance.
To find out more about this combination between practical and digital, befores & afters talked to MR. X visual effects supervisor Eric Robinson. He outlines the intricate VFX work involved, including roto and paint, and why MR. X mostly kept the practical eyes in the frame.
b&a: This seems to be another great example of MR. X doing really great CG work on the back of some amazing on-set practical work.
Eric Robinson: Yes, I find that when actors have something to interact with, their performance is better. I would rather augment a puppet like we did in this case—a puppet from the Jim Henson team—than to try to fix eyelines on three different actors in the same shot to try to make that work. You can really feel that engagement on set when the actors are working together and interacting with the puppeteers. It was their job to bring Larry to life. And VFX was there to support them, tune things up and help them when there was something that they couldn’t do as a puppeteering team.
For example, the puppet couldn’t run, but the interaction between Oliver and Larry, those are all in-camera together. The puppet was quite articulate. It had robotic fingers and hands with four people operating it. It’s quite amazing to watch, but the face itself was static. Larry doesn’t really speak much, but any real face performance was VFX augmentation.
b&a: Going into the show, was that always the intention? That the puppet was there for actor interaction and whatever performance it could get, but was there always a plan for facial animation and other augmentation?
Eric Robinson: Yes, we were brought on and in consultation with Amblin and the director, Jacob, in the early stages of the puppet development. We were on calls with Jacob and Hensons at the same time, just to really understand what the plot was going to be, and how we could help them get it up on screen. It was very collaborative that way.
b&a: What were some of the things that were tricky about your CG work here, say in terms of matchmoving or tracking onto the live action?
Eric Robinson: Well, witness cameras really help getting a solid track. Because this was done in Toronto, we had full access to photogrammetry for the puppet. Matchmoving with precise geo really helps to get the scene up working as good as possible; having that model, as our first step, and then augmenting to get the facial animation component. And, it’s the best lighting reference you could ever get as well. Because the puppet was dressed for set. The idea was to look a little slick so it’d be getting a wet down between takes to give glistening highlights and, especially in the dark, try to give some reflection for shaping. That made our job so much easier rather than just making it all up.
We’ve built a custom array of cameras for photogrammetry. It’s portable. It’s great. It can be reset for close-up or head to toe. And because we were working with a puppet that was modular, we were able to do hero photogrammetry for the hands or the feet for particular hero moments. Especially the hands for where he beckons. There were certain points where it was a fine line between mechanical and that stiff and awkward look.
When talking to Jacob about the character’s movement, the idea was he might be really stiff and broken and physically ‘clicky’ at times, like you wake up and you’re so sore. And so that sort of idea was running through our heads as we’re animating, knowing that you could feel the bones crunch and click—that Larry was that uncomfortable in our world and that was kind of the pain that was driving him to find a friend.
b&a: How did you approach the facial animation?
Eric Robinson: We went right down to the concept stage where we had a final model of the puppet. And then we had to design how big the mouth would open, what the eyes would do, how stiff the skin would be. The Larry model has scars on its face so the rigging of the skin motion would be really affected by the scarring, we’d get areas of rigidity and areas that are looser. That would affect asymmetry in the animation because of that.
A cool thing was, wherever possible—and it was a great tracking reference—we would leave in the practical eyes which were like marbles with a red backing in them. They would catch and reflect the light.
b&a: Where you had to do any full body animation or motion, were you just referencing what could be done with the puppet? Or was there any kind of post mocap or other vid-ref done for that?
Eric Robinson: Well, that was quite interesting. The puppet, when operated by the operators, was slow and methodical. There’s this long, long shot in the house. Larry had just come out of the TV set and he’s trying to find Oliver and mom and we’re kind of trucking back and forth. That was a great stalking moment for the puppet, and a lot of paint out for us.
Our model had the same rig as the puppet, the range of motion was well-defined by that. But part of Jacob’s vision was that Larry doesn’t really know how to be a biped. Larry comes from his own Larry world and being in our world is just not comfortable for him. So he’s got a very odd run cycle, which is kind of hands and feet. We went through quite a few iterations to try to come up with something that was eccentric enough to fit into the vision.
And that’s why we have motions where he’s like a spider. Whereas I thought initially, reading the script, I thought it might have been more like a gorilla fist down running. We did do some motion capture at the studio to try to get some starting points for that. It started off with something more like what a human could do, like mimicking a gorilla, and then ended up being something more like spider or a crab.
We did that with an Xsens suit, and I got to do some of the gorilla run myself. I don’t think any of my mocap for Larry ended up being used very cleanly, but I did get to be in Raised by Wolves, so that was good too!
b&a: You mentioned there was a lot of paint-out for this. Was the performer/puppeteer wearing some sort of bluescreen costume?
Eric Robinson: They had a black outfit and then blue as well. We were able to shoot clean plates, which really helped. But re-creating the wind through leaves behind and painting out a person was a big deal. It was a great accomplishment from the paint team. The clean plates would get you only so far because the atmos smoke levels would change, the wind would change direction, things aren’t blowing the same way. The rain would stop, so the leaves were dryer. A lot of tricky things to get around there.
The other challenge was in the scene when Mom and Oliver get under the bed. So you’ve got Larry’s feet and then six other feet around. And hardwood floors. A lot of reflectivity there. It wasn’t easy. And again, your clean plates only get you so far because you go from having a puppet and three people, blocking a bunch of light, to a clean plate. Luckily, the puppeteers were total pros.
