Image Engine explains the meticulous roto, matchmoving and tracking involved.
Early on in Lucasfilm’s Disney+ Original series The Book of Boba Fett, the title character (played by Temuera Morrison) and Fennec Shand (Ming-Na Wen) are ambushed by assassins on Mos Espa. A key aspect of the ensuing fight are the shields and weapons wielded by the assassins.
These were crafted by Image Engine, which worked with production visual effects supervisor Richard Bluff from Industrial Light & Magic on the sequence. With several assassins and the two heroes to deal with, Image Engine had to come up with a dedicated approach to adding in the shields and weaponry in an efficient way.
Here, Image Engine visual effects supervisor Robin Hackl breaks down the VFX prep work–including roto, matchmoving and tracking–to make the assassin fight possible.
b&a: With the assassin attack, what did they shoot for the scene?
Robin Hackl: They shot the whole sequence on location on an outdoor set. During the shoot, the actors just mimed having the shields. A lot of the foreground buildings immediately surrounding the characters are real. What we ended up having to do for the entirety of the sequence was extension work. Everything from a certain height up and in the distance was 100% digitally created in this environment.
b&a: The shields are a principal part of that sequence. Can you describe the design process and how you built something up that was not there at all on the day?
Robin Hackl: They gave us a few references, like the Gungan shields from the prequels. A few varying shield references and force field references were shared with us in the Star Wars lore. It couldn’t deviate from anything that we haven’t seen in Star Wars.
We opted to actually achieve the look 100% in comp, just to allow for faster iteration. It meant we didn’t have to go through a lookdev artist then to lighting then to render, then back to comp. We were able to give comp geometry and then we were effectively projecting the texture directly onto that geometry. It was a kind of 3D process since we still had to track the 3D shield to the assassins. We needed such an accurate track when the camera was circling around a crowd of them.
b&a: I wanted to make sure I gave a shout out to the artists behind the roto/matchmove/tracking work, because you obviously needed to separate all those characters out to make this happen, and then track these shields on.
Robin Hackl: There is a lot of upfront work that requires the brute force of every department. There was no department that was not touched in the making of that, and we had to take a unique approach to it because of the layering. In fact, our approach was fundamental because of the nature of how it was shot. We’re down low, we’ve got a group of six assassins plus the two main characters and everything always had to have its correct layer priority. It wouldn’t make sense if you saw shields on top of everything and you couldn’t visualize the depth of it.
It slowed us down initially because we had to prep it in a certain way so that we could get the client a quick temp for review and make sure that everything was working timing-wise. They’re interacting, they’re hitting shields, there’s an effect when the shield happens. Is their shield off or on? At one point, they start dropping their shields, so do they have a shield or not?
To get to that temp version, the very first thing that we started on was the matchmove and tracking. We needed a camera for every shot because we had to extend the city in every direction as a base requirement, so we needed a camera and then we needed to identify which characters had a shield or not. Let’s not forget, they also had an additional weapon. So both items needed to be tracked because this weapon was electrically charged.
The matchmove process was elaborate. Everything from, ‘Okay, this shot just needs a camera,’ to, ‘Okay, this shot needs a camera and all six of these characters need to be fully roto-mated.’ We realized, budgetarily, you can’t just roto-mate every character for every frame. It’s just not viable. Time and cost wouldn’t allow for it so we really had to narrow it down.
Our CG supervisor, Edmond Engelbrecht went in and annotated them by quickly drawing on the frame, where the shield and the weaponry should be. It was initially an internal thought process: ‘This guy’s shield’s on. This guy’s shield’s off. This weapon is on. This weapon is zapping somebody.’ This would be done with rough drawings in RV, just to annotate over top of it. What we ended up doing was sending that to the client and they were like, ‘This is great. This tells us a lot and actually now we have some feedback for you. That guy’s shield shouldn’t be on.’
We had to really be careful not to fall into the trap of overworking it, meaning roto’ing everything and matchmoving everything. Then once we had even a rough understanding and a buy-in from the client, we could actually go on with the matchmove. Once we had the matchmoves done, what that allowed us to do was place a three dimensional shield where the shield should exist.
If we had taken the approach of just roto-mating everything, in one sense it would make our life easier, but in another way, it would make our life hell. We’d have to go and rotomate every character. So instead we had to think, ‘How are we going to get this layering to work and visualize not only for output to the client, but internally?’
As a first step, we defined which shields and weapons were on. It allowed us to place the shields, and then roto started getting involved. And then we used it in a–I’m not going to say a unique way by any measure–but we had to take a very technical approach about it because it wasn’t just roto. It was roto that was placed on geometry that matched 100% the three dimensional positioning of the characters in space. So the roto existed and was layered accurately.
What that allowed us to do was, say for animation, for example. Now animation tends to use their standard rendering tools and they would have no way of holding out information from each other. But now they were able to have this as their output. Animation would have their red shields in there and their little temporary animated weaponry and they could put it into the scene in a very three dimensional accurate way, if they had accurate roto-mation for the arms. Now we could all see it together in a way that made sense.
b&a: I was curious whether you considered throwing any kind of AI machine learning roto at this problem for a quick and dirty pass at this sequence?
Robin Hackl: Not yet. We haven’t proven any of those tools for production at this point, but certainly those types of tools will be a major factor going forward. This one was difficult because these guys are wearing all the same uniform. Would it actually be able to pick them up off of each other, for example? It was also a lot of roto, something like 120 shots.
b&a: What I really liked about the shields were some nice aberrations and running lines and things like that on them. How did you accomplish this?
Robin Hackl: First there was the transparency side of it. Then there were all the complexities of being layered with ramps and animating bits–there’s these concentric rings that animate and have a bit of a vibe to it. And then there was the application in comp to do all the appropriate distorting of the image behind the shield to give it some sense that it was more than just refraction but also heat distortion, to give it a bit more of a menacing kind of quality.
b&a: I also liked the weapons that have an electrical charge effect. Were those Houdini sims or more hand drawn type animation?
Robin Hackl: We took a hand-drawn approach. I’ve often found electrical events really quickly become very procedural-looking. Not to say that they can’t be crafted and look awesome. There were two elements in play with the weaponry, with these electrical prods. One element was the ‘on’ state. It’s always on, it’s got this little electrical charge.
What we elected to do there was a combination of roto-mation or object tracking. 95% of the time, they were holding a ‘pike’, as their weaponry and we were able to object track that pike, stick a piece of geometry that represented the area of that electrical charge, and that would create a cycle that was always on. When it got more advanced and more nuanced, say when it was charging or when they were using the weapon on Boba Fett, it was more art-directed. It was more like, ‘More charge, less charge. It needs to arc really strongly here, but we want to see some residual arcs way over here on his body or on his backpack.’
I will say one thing: ‘electrical painting,’ if you will, when you ask an artist to paint electricity, it’s very nuanced. It’s very hard to do. It’s frame by frame animation effectively. But it has to have coherence. Electrical charges don’t just go on and off. It’s a very nuanced arc, just to articulate and stroke by stroke, frame by frame, make these electrical charges look convincing and move along the body in a meaningful and a convincing way.
b&a: There’s also the digital environment side to all this work. How did Image Engine build up the Mos Espa buildings and environment?
Robin Hackl: ILM built the entire city. They had a base construction of the entire city for the very large, wide establishing shots. And then they shared all that data with us so we had a significantly good starting point to then drive it from our end for our section of town.
We had the set construction there, which was effectively the first floor of the city. That’s built practically. Whatever we did had to sit with that in a photorealistic way. We’d basically start to set decorate it. We’re adding props, wires, things on the side of the road, and other things that can help tie it altogether.
