Behind the scenes of that final episode sequence.
The thrilling conclusion to season one of Hawkeye takes place largely in the Rockefeller Center area of New York City. That entire environment was recreated in CG by Industrial Light & Magic, complete with a 75-foot-tall synthetic Christmas tree liberally decorated with 50,000 dazzling lights .
Actors were filmed on a partial set build of the plaza in Atlanta, with ILM’s digital buildings, ice rink and surrounds allowing for just about any angle to be shot. Here, ILM visual effects supervisor Vincent Papaix, who worked with overall VFX supervisor Greg Steele, breaks down for befores & afters how the Rockefeller environment was crafted, including some special Marvel-related Easter eggs to look out for.
b&a: Where do you even start with building a New York environment?
Vincent Papaix: Early on, we knew that the on-set team at Marvel would go on location, shoot some plates, shoot some tiles, reference photography, scan, and Lidar the whole area. Working within the prescribed budget and timeline, sometimes we needed to be careful saying, ‘Oh, can we go full CG?’ But, we decided really, really early on that we needed to have a photoreal environment for any camera angle. We knew that Kate and Yelena would be jumping from high up on the building. Everything you see out there was fully CG.
We constructed the environment so they could move the camera wherever they wanted. It was a lot of work for the environment team, supervised by Matthew Lee and John O’Connell, but it certainly paid off in the end result.
b&a: I feel like ILM has had to create New York a couple of times. Marvel certainly has. Even before this shooting or even before you started some sort of capture or build, do you ever go back to ILM’s library and see whether you have this Rockefeller area in some capacity?
Vincent Papaix: Yes, every show is archived so that we can actually put it back online fi we’re working with the same client. So, it’s a process. It’s not like we have a folder and you simply click on it. It’s a lot of data, and files have to be restored from tape back-ups. We went as far back as Avengers, mostly to recreate that first sequence that the show starts with–the Avengers battle of New York. It was pretty fun to recreate that. So, what we did for that special sequence was, the brief was well, it needs to be like Avengers, just from a point of view we had never seen before. It needed to be seen through Kate’s eyes.
We managed to restore some geometry, but then we quickly realized that the Bishop house, the house where Kate lived, does not exist in real life. It’s a complete fabrication. So, we put it on top of one of the buildings.
Now, the environment created for Avengers was pretty impressive–Jeff White, our Vancouver head of studio, told me how the ILM team went and shot thousands of HDRIs at so hundreds of points across that swath of Manhattan. So, they’d start in the street, then from a skylift about 50’ off the ground, then they managed to get access to some rooftop. For us, unfortunately, we didn’t have that, because we didn’t have access to rooftops, and also it was a new space. So, we decided to go for a full CG approach, using a lot of photography as reference. And then we did something similar for Rockefeller Center. So, we matched what we knew, but then we went for a full CG render.
b&a: So, for the Rockefeller area, was Greg Steele and the VFX team able to do a full Lidar scan of the rink and the surrounding buildings?
Vincent Papaix: Everything, yes. They did a full Lidar and full texture shoot, and photogrammetry of the New York location at Rockefeller. And they also did the same thing on stage. So, that ice rink and just the entrance of the Rockefeller was recreated at the Atlanta stage. Despite doing a really great job at matching the real location with the set they built, one thing we realized that’s pretty funny is when you are in the street, you think everything is flat, right? But then you go into the 3D scan you notice there might be a difference in height of about two feet on a street. It’s not much, but when you’re in CG–where you tend to make everything perfectly flat–it doesn’t match the real New York city, which is not perfectly flat.
So, when we brought in the Lidar from the Atlanta stage, which was made more or less flat, we knew that we had to make some adjustments here and there to fit both environments. We needed it to be accurate to the Atlanta stage, because that’s the plate.Then we needed to make sure our CG extensions aligned nicely with the stage, but also seamlessly blended with the environment work of New York.
b&a: That’s interesting, because you effectively have three environments: the real one, the Atlanta stage, and your CG environment. And you’re just, I’m guessing, manipulating them, blending them, where you need to, to make it work.
Vincent Papaix: Yes. We took some creative liberties. But for me, it was really important to be photoreal. Everybody should feel that they are on this ice rink in New York. And actually, that’s another thing that helped us, was that we took a different approach for each shot. Sometimes we would keep a wall from the plate, or we would go full CG. Depending on the shot, we took whatever approach would provide the best result. When you have real and CG and you mix it up a lot, it really works. It’s like, ‘Oh, you know what, on this shot, the blend line is at this point. On this other shot, we are going to blend on the railing here.’ And you create a consistency, a messy consistency, but at least you don’t know. It’s not always the same line. So, your brain starts to accept everything.
b&a: Because you do end up having an entire CG build of this area, how is that visualized for artists at ILM in shot production?
Vincent Papaix: It’s definitely challenging to manage all that. All the credit goes back to John O’Connell, our lead CG generalist. He was taking care of the master files. We did the work in Autodesk 3ds Max and rendered in CHaos Group’s V-Ray. There’s a master scene, and then within the scene, you can say, okay, there’s reference files, so you can work in a given area, this way an artist could take care of the inside of a store and when they’re done, that’s a scene that we can reference in the master file and we do that section by section.
As we go, we’d say, ‘OK, we’re getting pretty close to the LEGO store, so we need to up-res this area so we have the one artist, instead of doing shot production, they will go in that area, up-res the model and textures as needed, and drop it back into the master file with John. Then any other shot will pick it up automatically.
b&a: One of the things I love about this Rockefeller sequence, too, is the lighting. It’s at night, so you get all those nice building lights and Christmas lights. What were the very specific lighting cues that you looked to really nail that New York look?
Vincent Papaix: Well, it’s something we almost got, not for free, but let’s say by analyzing the tile photography. That complexity came by looking at the photography and thinking, okay, what’s the main impact? In the ice rink, you had those blue lights, there were some massive lights that in the real location are there and had been recreated on the stage in Atlanta. About 80% of the lighting was coming from those big area lights on the ice rink.
But for everything else, it’s really about matching the tile photography. We didn’t fake anything. Everything is there with a light that bounces and ray traces all around. It was heavy to do, but you get the return.
b&a: Another thing that’s incredible is the detail, down to the storefronts and the fire hydrants. But I feel like it is also moving cars and crowds that give a lot detail to the scene. How are those accomplished?
Vincent Papaix: The cars were keyframed by the environment team. We have a library of vehicles that we can tap into. We try to be true to New York City, so we put the ubiquitous Yellow Cabs, various cars, trucks, and SUVs.
For the crowds, we leveraged ILM’s proprietary crowd system and our crowd supervisor, Phil Rouse, did a fantastic job at gathering all the data. We created a bunch of agents from scans from the set, but also from our library, and we made sure that it looked wintry, so we outfitted them with the appropriate coats, hats and warm clothing. They even had branded shopping bags,, as if they had been doing their holiday shopping around the city. We didn’t do any motion capture for this sequence, but we used our motion database where we have people walking around. Some people even taking selfies, things like that. What was a bit more custom was to add the hand bags and shopping bags. We attached the bags to the hands and made sure that it was doing the proper motion for the arm.
b&a: The Christmas tree is amazing, as well. What did they have on set in Atlanta? And then, how did you start the process of building something that was ‘kind of’ CG foliage, but it’s something else as well with the lights, isn’t it?
Vincent Papaix: The tree was definitely one of our main challenges, mostly because the real one has over 50,000 lights. It’s also a spruce tree, about 75 feet tall. We looked at the needles of the spruce tree, which in this case has them attached to each other before going to the branch. We created a few variations of needles and scattered them–it’s like millions of needles scattered throughout the tree. Then we put the lights on and matched them to the real one in terms of the light bulb, the plastic connector and the cable connecting the bulbs.
On set, they replicated a portion of the tree so we had the first four or five feet to work with. Ultimately, we had to make it larger at the base when we did the shot. But we were still blending the plate and our CG tree where we could. So, we extended the tree at the base, and also anything that’s more than five feet up was full CG.
Then we had the tree on the ice. They shot a practical tree falling down in Atlanta, but it was quite a bit smaller compared to the volume of that gigantic tree. So, we ended up replacing it but the practical version served as a great reference for the team.
When Clint is in the tree, on set they had recreated a portion of tree with real branches. It was five or six branches with lights around them. We tried to keep as much as we could from the plate, but we augmented it with CG tree parts. We had to create several layers of far background and midground tree, and then we’d sandwich the plate and have a couple of foreground elements of CG tree to sandwich him in and blend it together.
b&a: When you were building the fully CG tree with lights, did the fact that it’s thousands of lights, I mean, did that scare anybody in terms of wrangling the sims and the renders?
Vincent Papaix: Yeah, I was part of the scared ones! But it’s part of the challenge. We have a substantial render farm at ILM, which is nice, but even if you have one of the biggest farms, you still never have enough resources, right? The good news is the tree was handled in Houdini for all the scattering of the needles and lights, then it was cached, and then it was pushed into our Katana / RenderMan pipeline. And working with Pixar as well, we are able to leverage a lot of good tools in RenderMan, and there’s some really smart ways to manage the RAM you need to cache all that and render it.
We still did full raytracing, although luckily if you look at those Christmas lights, they’re not going to light 20 feet away. They’re going to light just a short distance away. So, we found some optimizations that way. We’d raytrace everything, but we’d limit the distance and also limit the bounce to find some optimizations.
It was definitely heavy, not just the lights, but also the flickering. That’s one of the worst things for a renderer–you have lots of tiny lights, and on top of the tiny lights, you have tiny needles that occlude the lights! So, to resolve the noise in there was not fun. We cracked the code, though, and I think we managed to find the right recipe to deliver a good image.
b&a: When I was watching it, I was like, ‘Ah, I wonder how much they built of it?’ And of course, you mentioned the partial build that they did do. I was also thinking about when the camera and the arrows are moving around, and you see all the light streaks from the tree lights. I just thought you really dealt with those light streaks and blurs so well.
Vincent Papaix: To come back to what I was saying about trying to blend real and CG, the tree is CG the whole time. We realized that the tree, even in references from the on-set VFX team, depending on the angle, depending on the exposure of the camera, even the focal length, the tree would look very different. So we thought, ‘We need to have some kind of consistency.’ Even if in real life, you look at it and I show you five photos, and they look completely different, and I’ll tell you it’s the same tree, you will accept it because it’s a photo. But if I do that in CG, you’ll say, ‘Well, they don’t match.’ So, what we did was an exercise of making sure it always felt like the same tree.
b&a: There’s some crazy shots with it of the tree falling, but there’s also some shots that are just ‘normal’ shots that I now realize are fully-CG. Like seeing the tree and lights showing through those flags around the rink.
Vincent Papaix: So, that was interesting. Those flags, the gold and silver flags, they are actually transparent. Greg Steele shot some reference for us, and we had some reference where there’s actually some bright lights behind the flags, and we realized that they were actually transparent. Our approach was to match what we saw on location in New York. I mean, they look really metallic. The specular is really sharp, and they’re actually really, really thin. So thin that you can actually see through with a bit of refraction, so we carefully matched the shading response to make sure that we had the right response and stayed true to the real location.
b&a: That’s awesome. Is there anything else you wanted to say about the work?
Vincent Papaix: Well, if you look closely at the LEGO store in episode 6, there’s some fun Easter eggs in there. We told Marvel, ‘Maybe we can put this in there?’ and they loved it. So, that was quite cool. If people want to look closely in the LEGO store, there’s some interesting LEGO in there that we put. It’s pretty quirky, but it’s always cool to see.
And it was the same thing for episode one. Marvel was saying, ‘Can we include Iron Man? Can we have people from Avengers?’ We managed to actually hide one Avenger that nobody has found yet. I won’t tell you where it is in there, but there is one Avenger somewhere in episode one.
