I mean, have you ever *tried* to animate a pianist playing the right notes?
In Pixar’s upcoming Soul, directed by Pete Docter, one of the challenges the animation studio faced was depicting its main character Joe (Jamie Foxx) playing the piano, believably. That is, animating his fingers to hit the correct keys for the music he was playing.
It’s certainly something that can be tackled just using video reference—which Pixar’s animators certainly did—but the studio took things a step further by implementing a workflow that went from the music of jazz composer Jon Batiste, to MIDI files of those performances, into Houdini, and then into Pixar’s proprietary animation software Presto; all to determine the piano keys that needed to be ‘played’ and therefore how the fingers and keys would be animated.
befors & afters was able to ask Pixar animator MontaQue Ruffin and character modeling and articulation artist Jason Davies how this process worked for the film.
Starting with reference
Ruffin says the first step in the piano animation process was establishing the right kind of physicality for Joe. “A lot of it was thanks to Jon Batiste, who was our direct resource and also part of our cultural consulting. Not only did he score some of the music that you hear in Soul when Joe plays piano, but he would come to the studio and play performances for us.”
During these performances, a Pixar video crew would record footage of Batiste’s fingers from multiple angles. This vid-ref was used heavily for the final shots by the animators, who followed the reference frame by frame by translating the performance to Joe.
“In addition to that, we had some technical breakthroughs,” states Ruffin. “We were able to take Jon Batiste’s MIDI audio information and import that into Presto, our 3D software. The video reference of Jon playing the piano let us pause on any frame and see where those fingers were landing at. However, some of those frames were kind of blurred out because the fingers, they’re just going that fast.
“So, being able to import that MIDI data helped us immensely because it meant we could ‘project’ each of the keys that Jon Batiste was playing, and use this as another visual aid as we roughed in our animation. Once we had that hand placement, we were able to then layer in the fingers, because if the hand placement was off, the fingers were not going to be in the right position to hit the keys at the right time.”
How this MIDI workflow worked
The idea to provide Pixar’s animators with that ability to visualize the right keys was inspired initially by an antique ‘automatic’ player piano that is at Pixar’s campus. Seeing the time-consuming effort that was involved in match-animating the vid-ref, the technical team at Pixar wanted to help automate the process. “Something like a player piano,” relates Jason Davies, “with the keys moving with correct timing and force, would greatly speed up their process and allow them to focus on the character’s performance.”
To do this, Davies set up a system for using Houdini to read MIDI (from Batiste’s performances) and to write very basic animation files that Presto could understand.
“The Editorial department would edit shots from the film and the corresponding MIDI files, both left and right hands, at the same time to make certain everything lined up properly,” Davies explains further. “A copy of these MIDI files were then saved to a special directory named by shot number. MIDI files in this directory would be read and processed every night, using Houdini.”
“Then for the Animation Department,” continues Davies, “I wrote an animation tool in Presto that would read the animation directory and display all shots available. Once a shot was selected it would author the performance animation onto the piano rig, i.e., keys moving and color coding for the left and right hands. The color changes were for display only and were ignored at render time.”
Getting the flair right, too
While getting the animated Joe to hit the right notes—literally—was one challenge, another became giving the character an appropriate style, something that came from both Batiste’s playing and from Jamie Foxx’s personality.
“There is a flair to Joe,” discusses Ruffin. “Not every pianist plays the same way. What the video reference allowed us to do was just have that foundation, and then it was up to the animators.”
“A lot of it was about timing and spacing,” adds Ruffin. “When you framed through a motion like that…frame one, two, three, four…the distance between frame one and frame two, that’s what you call spacing. If you hyper-focus on the spacing of the fingers and how they whip or or flourish as they’re striking the keys, you can have a lot of control in getting the right flavor in there.”
