‘Wow, that’s a lot of ownership. That is rare for us in visual effects.’

How Scanline VFX helped shape a number of pivotal scenes in ‘Eternals’.

We don’t always talk about the significant design work that visual effects studios get involved in when completing their final visual effects shots on a big film. While they may often be working from approved concept art or early previs, many VFX studios are also regularly called upon to help craft entire designs for characters and worlds, especially as a film evolves.

Jumping headfirst into many design challenges was Scanline VFX on Chloé Zhao’s Eternals. The studio worked with production visual effects supervisor Stephane Ceretti to help conceive and execute several key moments in the film. These include Sersi (Gemma Chan) learning her true origins as she speaks to Arishem and visits the World Forge in the space void, the delivery of a ‘big bang’ moment, and also showcasing a Deviants versus Eternals fight in the streets of Camden.

In this frank discussion about the work, Scanline visual effects supervisor Jelmer Boskma breaks down the design work and the final VFX crafted by the studio for the film.

b&a: I wanted to talk to you about the art direction side of Scanline’s visual effects, first. Obviously it’s quite common for many VFX studios to contribute towards the design of sequences, but how did it work on Eternals for you, especially for the World Forge?

Jelmer Boskma: Well, at Scanline, we don’t have an official art department. You’re sort of speaking to it right now. Besides myself we have been working with a number of freelance illustrators over the years when the awarded work called for additional design help. We’ve only just started to get some people in-house now, which is exciting. On Eternals I had help from a wonderful illustrator in our Munich office, Benjamin Ross. Together, we explored designs through illustrations mainly. He’s much better versed in 3D than myself. I like to just go into Photoshop and put down some paint, because it’s quicker, and for me at least, allows me to communicate ideas, and that’s really how it started. We were still shooting, and [production VFX supervisor Stef Ceretti] started getting us to look into some designs.

The previs we had depicted some of the beats for the sequence, like the World Forge and Sersi discovering who she really is and Arishem explaining this to her. But much of the previs changed quite severely. So the design of that, of the World Forge itself, both the exterior and the interior, was carte blanche for us. We had a couple chats with Chloe. She was very adamant about, ‘We want to instil the spirit of Jack Kirby into this. Look at the comics, and look at the way he draws things, which is this visual language of hard graphic shapes mixed with alien forms.’ You know, it’s not messy, but it’s not sleek either. There’s a lot of texture and grit to the way he designs, and of course, he has incredible bold color choices. Those were things that got thrown at us.

Stef suggested, ‘Perhaps we can use the Domo, which they had a design for.’ This is a green triangle. ‘Maybe we can use that as a base. We can build a geosphere from a few of them,’ and that’s where it went.

We had been doing some tests on other shots which is where the tentacles came in and all the golden filament motifs. Much of that was based on work that was getting done on other parts of the movie. We were starting to find a language for these transition moments, which are these gold, graphic shapes where we blend between scenes and shots. We tried to funnel some of those more resolved concepts into our paintings, sketches and 3D maquettes for the World Forge. We just first needed to find the language, and then tried to instil some sense into it. At some point, I mentioned that, ‘Perhaps we should have these tentacles absorbing energy from the outside, feeding this core in the center.’

We thought maybe that’s what powered this entire thing. Ultimately, it took a long while to find the look. Chloé was very sensitive to all the cliché cinematic fantasy illustration tropes, such as too much atmospheric perspective or muted color schemes. She wanted the interior to be perfectly clear and definitely kept us on our toes for that stuff.

As we started to find the architectural base for the World Forge, there were some pretty wild ideas regarding dressing this virtual set. ‘Maybe we should put some vegetation on these surfaces similar to the plants we see in the Domo, because all this stuff is partially organically grown’. I won’t lie that at times my mind went to ‘Oh my gosh, plants in here now? How does that fit in, and crystals?’

But this is all part of the process of finding something new. I think it took about a year, to be honest, to get to the point where it’s like, ‘That’s it,’ mainly to get out of our own comfort zone, I think, but also to find something just truly unique. Ultimately it was fun and rewarding because it was such a direct collaboration. Obviously, for myself, when I look at the World Forge, it’s just like, ‘Wow, that’s a lot of ownership. That is rare for us in visual effects.’

b&a: I think it’s very rare actually, although clearly someone like Stef and yourself can make that happen.

Jelmer Boskma: Stef has been with Marvel for such a long time. He knows how the process works. But at the end of the day, it’s a committee. There are a lot of clever and very invested people that have opinions on how things should look and why. There’s a fair amount of brainstorming that goes on behind closed doors to which we’re not privy. There’s quite a number of people to essentially sell our shots and our ideas to. With Marvel, specifically, finding consensus on an idea is sometimes half the battle, from a rudimentary conceptualization point of view.

There will be conversations, like, ‘Well, why do we use these colors?’ and, ‘What is the function of this place? Do we need to tell that story?’ A lot of that was not figured out. The wall of memories moment for instance–we had come out of the pandemic hiatus. We had been shut down for two months which gave Chloé time to think and she came back with that wild new concept for the sequence. Originally there was more interaction with what we called the ‘Void’ Deviant, this bird-like deviant that we built, telling the story about how Eternals and Deviants are related. This later found a home in the Centurii six scene.

There was a version where Arishem shrunk himself down and essentially was like Willy Wonka walking through the World Forge. That got lost, but the wall of memories and all of that stuck. Even though it was something that was conceived much later. The entirety of the World Forge was photographed during a separate pickup shoot, long after principal.

b&a: For the World Forge, what could they shoot with Gemma Chan?

Jelmer Boskma: Well, she does deserve a fair bit of credit for this because, not much. It was a blue void all around and a light source. By the time they came to shoot, Chloé had seen some paintings, and we presented designs through Stef. They’d say to Gemma, ‘You’re going to be looking at, well, this-ish. There’s this replicant coming down, and then there’s this wall coming up, and now you’re touching a copy of yourself.’ They had a mannequin, just a gray mannequin of her height. There was nothing there. Those plates were very empty, but all props to her because I think part of why the sequence works as well as it does is that she did manage to land the emotional beats, and takes the audience through the feelings of confusion and the sense of wonder and fear that Sersi goes through in that moment. I mean, it’s not uncommon for actors to be asked, ‘Okay, pretend and go in this empty space,’ but I think she did exceptionally well given what she had to work with.

b&a: I’m guessing a lot of the exploration remains just that, exploration.

Jelmer Boskma: Indeed. Earlier on, there was a much longer version of the big bang moment starting with a close up on Arishem’s hand as he’s creating a sun and a supernova. Then, eventually, we end up on a dying sun and a planet, an Earth-like kind of planet, turning ice cold and dying off. The idea used to be that the camera actually kept flying into this alien planet, Centurii Six, and we see it being built up as a time lapse, not unlike that incredible timelapse shot in Darren Aronofsky’s Noah. We plunge into the water. We see the landscape grow around us. Then we come out, and we see creatures and life flourishing, and it’s all a giant alien planet. We storyboarded that. Previs’d a large part of it and did a bunch of keyframe paintings, but the idea changed.

b&a: Oh no!

Jelmer Boskma: It does happen from time to time, but it usually happens much earlier on in tandem with the script being refined. The art department and the visdev group at Marvel are the ones that tend to explore wilder ideas. Usually, that stuff is sorted by the time we step in.

b&a: Say when you’re designing the World Forge and the wall of memories, were there any particular challenges when you started getting into the actual build?

Jelmer Boskma: It was a pretty standard build, I would say. The trick there was finding the language of the materials first and foremost. Asking questions like, ‘How close can we get this to real world references?’ We settled on crystal and this very translucent, I’d say, coral, slightly denser crystal, perhaps.

We would build the whole thing in Maya, do the sculpt pass in Zbrush, and then start putting shot cameras in and say, ‘Okay, where do we need to think about extra detail?’ It turned out that to get the right look, we ended up scattering hundreds of thousands, if not millions, of individual crystals in all these little cubby holes in that wall. Those were individual objects. Then, well, they’re crystals. They’re pretty damn translucent. How on earth are we going to render millions of crystals? We had done something similar for that, not to that extent, but it was actually in Ant-Man and the Wasp. The restaurant fight scene in that movie had a crystal chandelier, which ended up being destroyed. We had some close-up shots of Wasp flying through that. We had all these little crystal beads. So there we had to find ways to actually physically get it to render, because at a certain point, there’s too many rays to trace. There’s a limit to that, but we learned a lot from that, and that helped in how to effectively push that through and also build to camera really, because you can put the detail where it’s needed.

b&a: How did you approach the creation of the big bang sequence?

Jelmer Boskma: There’s an enormous amount of effects there. The shot was really cut up in three parts to allow different compositors to take the helm at different moments in that shot, and stitch it together, and have that overlap and make it, well, seamless as one shot, but the effects work in that moment specifically, I think, was really interesting for many reasons because it’s a lot of grand scale, surreal stuff. We start with essentially a celestial big bang, and then go into the creation of the universe with dark matter forming webs and the creation of protostars. We tried to make sure we could instill a little bit of logic and make it presentable..

We had fairly decent previs, but it was generic in what it was depicting. We tried to tell that story, again, it was one of those things where you just scavenged the web. It’s like, ‘Alright, astrophysics, how was the universe born?’ Let’s brush up on that, and try to instil some logic at least to cling onto as these things take shape, and that cloudscape of nebulae, just pushing it through like, ‘How do we create something like that with nice, whimsical motion, and seeing stars being born inside these galactic clouds, and how do we art direct that specifically to form an aesthetically pleasing shots, where the composition works in our favor, and then have motion on top of that?’

And how do we render all of that? There were so many moments in that one shot that required really creative thinking. We actually ended up also applying NASA footage, like photographs of neighboring galaxies, and translating those directly into three dimensional point cloud data to have physically accurate galaxies for our backgrounds. They’re peppered all throughout that shot. It was a tremendous effort from FX that went into that to orchestrate all those components that make up the big bang sequence, and building the Celestials as well was one hell of an effort because of both their scale and alien design.

We knew we were going to have to get pretty damn close on Arishem. If we wanted to put the camera on his ankle, we could. It was all there, but these other Celestials were also built to an extremely high level of detail, and developed in-house, including Tiamut, the big golden guy, which we then shared with Weta Digital. We were given a single sketch from visdev for each celestial, giving us a great base to start exploring from.

It was a lot of fun to develop the look of those guys, finding the language, really, of the Celestials. We were asking ourselves, ‘Are they metal or rock or some strange mineral? What are they made of, and where does the light come from, and how do we make them all feel unique and identifiable and instil enough personality?’

b&a: And then of course Scanline also did the Camden scene where the Deviant appears. How did you deal with this?

Jelmer Boskma: Obviously, it was a very different type of work. For starters, we had plates to work with. That was different. We had a wonderful Kro asset from ILM. They did the majority of the Deviant work on the movie. We did some dev early on on translating some of the visdev concepts in the form of our a tiger Deviant, which originally had a larger part to play in a sequence that sadly got cut. It did however end up in the movie. We gave that asset to ILM, and they used it in one of their Babylon fights.

For the Camden scene, we inherited ILM’s Deviant asset, matched their lookdev, and rigged it, which is no mean feat because these things are complex creatures, so we rigged it and got it ready for muscle simulations and just the usual creature work.

Then we had some interaction with water to worry about and interaction with sand. The quicksand was interesting, finding the look for that, because it’s not really quicksand. It’s more like Hollywood quicksand, where it’s actually very granular and dry. Real quicksand is liquidy and just plain bizarre.

Then we get to the illusion moment with Sprite, which was really cool because we knew we were going to need a lot of copies of our actors. Motion control was always on the cards because it’s an effective way of doing that. These passes were filmed later on a stage. I was skeptical at first simply because of the way the Camden location was lit, which was through many different, smaller brightly colored light sources, but very specific lighting. I thought, ‘You’re never going to match that again on a soundstage, no way. And even if you do, there’s so much small reflective detail affecting the tone of the skin, which as you know, is highly reflective.’

I ended up pushing a bit for Scanline as a whole to embrace this as an opportunity to really push our digi-double pipeline. We got, I think, phenomenally good detail into both the Sersi and the Sprite digi doubles. And sure enough, they ended up right, front and center in some shots, because we didn’t have the right take from the motion control, or quite frankly, the lighting on the CG puppets looked a bit better.

The motion control passes were done in LA during pickups. With the camera track data that we provided for the shots lensed in Camden, driving the motion control camera. The passes turned out better than I could have hoped for and only required minor grading adjustments in most cases. The funny thing was that our Sprite actress, Lia McHugh, well, it was a year apart in shooting and her character has a rather peculiar hair style, so she was wearing a wig for those motion control passes. That wig was not loved by everyone, so we ended up, for some shots, either grading the wig or replacing it completely with our CG hair on the MoCo pass. It was a complete hybrid.

b&a: And so, for this you also did the digi-double work for that scene. Did this mean that Scanline took the lead on building a CG Sersi and a CG Sprite?

Jelmer Boskma: We did, which is not uncommon. Marvel wanted to make sure that we were comfortable doing facial animation to quite a high standard as well. We worked on some tests, which we presented, and that convinced them. I’m very happy that we had the opportunity to actually flex our muscles a little bit outside of the destruction and water simulation work that Scanline has mostly been known for.

b&a: I actually really like that Scanline is not just the ‘water sim’ place anymore, not that I want Scanline to stop doing that! Tell me a bit about what you actually receive in terms of digi-double assets from, say, a photogrammetry scan.

Jelmer Boskma: Marvel’s got a pretty great standardized process for that. They work a lot with Clear Angle on that. It’s just your big camera array rig. The actors are photographed in all their costumes in which they appear. We get that data back as well as a preliminary solve, which we then work with and clean up as a starting point. We then go through a more manual process to translate as much of the captured data into a clean puppet that we can rig and animate.

You know, when I started out in the business, I was modeling, and all this wonderful scan data was not at all common! We had turnaround reference photographs, and you had to line up your Maya cameras with that, and get going, mostly by eye. Now, I always give the guys a little bit of crap for that, like, ‘You’re spoiled with your scans,’ but it’s super useful. They’ve gotten so good, and it helps enormously. It just speeds things up, but there’s still a fair bit of manual care and elbow grease needed to really translate all the detail for our hero characters.

A lot of the work goes into those microscopic details, and it’s getting the skin pores to break up the specular, the oiliness, the reflective qualities of the skin correctly, and have a layer of peach fuzz to aid with that, and get the glancing angles to look correct, and just capture the likeness on different expressions, and how do expressions move from expression to expression, and how lights scatters through the skin. There’s a lot to balance on digital humans, and that’s an ongoing effort.

There’s a wealth of knowledge, but still, it remains one of the trickier things in VFX as a whole to successfully do that. The other thing we got from production was essentially a set of photographs or range of motion of Gemma’s face, so we could match that. I believe we also even had some Medusa data from ILM, where they captured some of the movement of her face, which was useful as well to get it just right. The funny thing is, we built a puppet, and we based that all on data that was captured, so it were her expressions and facial muscles firing, but Gemma was not the one driving her own face in that sequence.

Because of the pandemic, we couldn’t get her, so it was one of our own animators who got dotted, markered up, and we solved that facial track onto the Sersi digital puppet. It’s really interesting because so much of the likeness of why somebody looks the way that they do is because of the way they move their face around.

The way in which these muscles get triggered, or these shapes get activated in the order and the timing is just as big a component of the whole trick as are the actual shapes that work on their own. But now, we had to channel our inner Gemma Chan, and get that likeness. Thankfully, it’s not like we were on extreme close-ups, although I’m confident we would’ve been able to work something out for that. It’s our own facial tracking software on one of our animators that is driving the facial performance, and then a little bit of tinkering on top of that by hand is what made it work. COVID filmmaking, man!

b&a: I did a story about the re-release of Justice League where Scanline had done some facial animation remotely for Martian Manhunter.

Jelmer Boskma: That’s right. That was very similar, how we approached both of those challenges. But there, you had the benefit, well, not quite a benefit, I mean–it’s a bit of a creature. You have a little bit more wiggle room, and this was straight-up Sersi. You have the real Gemma in the plate with her digital copies. They’re literally side by side, so people can compare if they want to. You can freeze the frame, and literally compare all the Sersi’s in the scene. That made it extra tricky, but all in all really fun.

b&a: The other part of that sequence, which became a total trailer and TV spot moment, was the bus being turned into rose petals. Tell me about the design but then also the execution of that shot.

Jelmer Boskma: Well, the bus was never there, so the bus was completely digital in those shots. It’s interesting. I’ve seen comments on the web like, ‘What happened to all the people on the bus?’ Trust me, we made sure it was the last night bus and there was ‘Not in service’ on the front of the bus. Feel free to check! We start with a wide shot where Ikaris and Kro in their fight are tumbling towards the bus and crash into it. That’s where we ran a bunch of rigid body dynamic simulations to crumple up the bus, and shatter some of the glass. The flipping over of the bus was then driven by animation, keyframing that momentum correctly to get that swing upwards to happen. Of course, there’s a lot of art direction needed there. We tried to do it as believable as possible, but we had to take some artistic licence too. I’m not sure if we could get our physics past Neil deGrasse Tyson, though…

We got the bus up in its destructed state and scraping along the ground. We have one shot, which is interesting because it’s this over the shoulder moment of the bus driver. That again was a pickup shot. It’s a real bus driver in that shot, a real London bus driver who plays that part. We got him back for that shot, and filmed him in a partial set build of a steering wheel and a console, but that’s on sound stage. So for that specific shot, we had to rebuild the environment, the Camden Street environment around him, which was done with all the data that was captured on the day.

We had some LiDAR and tons of reference photographs, but we now needed to light it to match the front lit headlights look from the point of view of the bus driver. That’s the moment where they’re tumbling towards it, a little bit before they crash and actually have that bus flipping over. Then as it’s sliding and scraping towards Dane and Sprite, Sersi steps in the way, and uses her powers to transform the whole thing into flower petals.

To be honest, at this point, whether it’s fire or water destruction, it doesn’t matter really for the FX team at Scanline. Millions of flower petals? Sure not a problem! It is mostly a matter of getting the dynamics right and making the behavior feel interesting. We ended up getting some really interesting swirly motion to become part of it. A challenge was to correctly make them feel like flower petals in the way that light scatters through each individual petal, and gives you a sense of depth and clustering just by stacking and self shadowing.

You get darker moments in these larger groups, but in general, getting that right, and then also, ‘Okay, now we have hundreds of thousands of flower petals floating around,’ that’s going to affect the environment slightly too, all this light bouncing off and refracting through these flower petals. We did some subtle grading work on the plates to warm that up a little bit, and with, I think, some rotomation on Sersi, because we wanted to make sure that she was properly held out, so that we could still get a glimpse of her, and knew where she was in space.

For all of that, actually, it was pretty helpful because we had LiDAR data, which we cleaned up and used to hold out the environment pretty accurately. Allowing us to have the flower petals interact with the physical street, and trash containers and all of that, and then use that same data in compositing as deep data to then help them integrate all these flower petals correctly into the scene, and have the plate sit where it should. So it was just a matter of capturing the right data on the set, and then applying that in post to make that whole magic trick work

The one thing that was tricky was the edit. I think at a certain point in editorial, a different order of shot was suggested, in which case, it’s about lining up these simulations across multiple shots. We were also left with the residual flower petals that are now left on the street. We had to track continuity for those, and at times, just peppered them in here and there to make a shot look more interesting. When Kro comes back and lunges at Ikaris, it was cool to have him splash through some of the flower petals, or have his foot dragging some of them along. It just was an interesting additional dynamic element in a lot of these shots, and gave us just an excuse to add some more realism, which was great.

b&a: There’s even a nice little moment, isn’t there, where Kit Harington’s character catches a petal. I just assumed that’s a digital petal or a CG petal.

Jelmer Boskma: It is. It’s a CG petal. All of those are. He catches it, and just looks at it, and that’s just a matter of tracking it in there correctly. The whole thing is a fun big moment. And it really popped up left and right to help sell the movie. It is always flattering when a studio decides ‘These are cool moments. It’s going to get people to the theater’. Seeing your work used for that is always nice.’