“I went off, and six months later I came back with this algorithm.”
Paul Lambert, now a two time Oscar-winning visual effects supervisor for Blade Runner 2049 and First Man, and also nominated this year for an Academy Award for Dune, was working as a compositor at Digital Domain in Los Angeles back in 2004 when he somewhat cavalierly suggested the development of a new keyer for the studio’s groundbreaking compositing system, Nuke.
At that time, Nuke was already legendary in the visual effects community but for several years had only been available to use inside Digital Domain. In 2002, a spin-off company D2 Software had been launched to sell Nuke to the outside world. Lambert thought it still needed a good keyer.
“They were basically offering Nuke without a keyer,” recalls Lambert. “We were in a Nuke user group meeting and the whole keyer thing came up and I sheepishly put my hand up and said, ‘Look, I understand how these things work,’ because I had figured out some keying things when I was at Cinesite back in London. I said, ‘Well, look, give me some time and I’ll figure this out.’”
“I remember the guy running the group looked at me and must have thought, ‘Who is this guy?’ But I went off, and six months later I came back with this algorithm.”
That ‘algorithm’ was the design for Nuke’s IBK keyer. And it’s a keyer that still exists nearly two decades later inside the compositing tool.
A quick IBK lesson
So, what is the IBK keyer? The very non-technical answer to that is that the ‘image- based keyer’ is a proprietary keyer in Nuke that typically deals with classic bluescreen or greenscreen plates (that need keying) by recognizing that these plates do not always have uniform color coverage. We’ve all seen uneven blue and greenscreens; that’s one place where the IBK Keyer can come in handy.
The IBK keyer does not set a single color-pick as the basis for the matte extraction process like some other keyers. Instead it generates a ‘clean plate’ using the pixel values of the compositing images that preserves the color variations in the plate. This is then used to create the key and therefore generate a matte for the image you are looking to extract.
In Nuke, the IBK Keyer consists of two nodes, the IBKColour node as the first step, and the IBKGizmo node, which is the node that pulls the key. IBKGizmo does not actually perform the composite; for this you feed the output of the IBKGizmo node to a compositing node.
Before IBK
OK, so now you know what the IBK Keyer does, what exactly led to Paul Lambert suggesting its development at Digital Domain? Lambert says it was very much based on his previous experience at visual effects studio Cinesite in the UK.
From the late 1990s, Lambert had been working in Cinesite’s commercials division on Flame and Inferno (meanwhile, Cinesite’s film division had largely remained a Cineon shop). He composited on numerous commercials and on a number of TV mini-series and eventually films, largely self-teaching himself the real-time compositing tools then sold by discreet logic, and picking up key skills about image processing. One particular interaction then changed the way he thought about compositing forever.
“It was one particular day in which Simon Eves, who was the head of R&D at Cinesite at the time, came down and I had a question about something or other, and Simon started to explain to me how an image can have a value over one. I just sat there in silence because he’d just literally blown my mind. It was like, ‘What do you mean a RGBA image can have a value over one…!?’ Flame and Cineon, at that time, only exposed the user to values between zero and one for all images.”
Meanwhile, Cinesite began shifting its compositing solution to Nothing Real’s Shake (later acquired by Apple). Lambert says he started to use Flame and Shake concurrently to find compositing solutions. Shake’s node-based workflow also offered a new way of thinking about compositing.
“I could literally put it down to two nodes inside Shake which define how I came to understand compositing, which were the Plot Scanline and the Gradient nodes,” states Lambert. “Basically you could draw a grad across the screen and then you put a Plot Scanline on it and then you put down other nodes in between and you’d see how that scanline changes. This basically defined how I worked out every single node inside Shake, which I then did inside Nuke as well, as I tried to figure out how that visual language of compositing actually came together.”
For example, adds Lambert, “you could draw a simple shape on a black background in white and you blur it and then you invert it and then you multiply the two and then you end up with an edge. It’s like, ‘OK, that’s one way to do an edge detect.’”
How IBK came about
With that Cinesite experience behind him, and having heard and seen what Nuke was capable of, Lambert specifically sought out employment at Digital Domain in 2003–it was then the only real way to get an opportunity to work with the compositing tool. After about two years at the studio, the conversation on keyers in Nuke arose, and Lambert suggested he could put one together.
“It took me about six months and it went through various iterations, but in the end I presented a paper to DD with the algorithm, and, honestly, it’s such a simple algorithm that, to this day, I am surprised that nobody else has done something similar to that!”
What Lambert devised for his ‘simple’ IBK keyer came from, he says, that idea of the Plot Scanline node and “putting different shapes onto a blue background or onto a green background and blurring those shapes and getting that color to bleed into the blue. It came from all the bits and pieces that I had learnt about channel differences.”
Lambert also observed at the time that other keyers and the nodes related to them had become ‘stacked’ with sliders and tabs. “What I found was that it was rare to find somebody who knew exactly how to explain all those sliders. Keying had become more of a trial and error thing. What I really wanted to do was to make something very simple, where compositors could expand on the actual idea.”
Indeed, that simple approach drove Lambert’s design for the keyer, and the original IBK keyer nodes–IBKColor and IBKGizmo–have essentially remained the same in Nuke ever since. It’s also always been possible for users to ‘open up’ those nodes to explore further, as Lambert desired.
“Anyone who has ventured inside the IBKGizmo node will see that there are four inputs to the IBK node rather than the customary three, which you get on the outside of the gizmo. I wanted to expose the algorithm to compositors so if there was a compositor like myself who was into the technicalities of these things, they could then go on and expand the algorithm out. Originally, I just wanted to keep the node, the original IBK node, as a group, but for commercial reasons you couldn’t really do that. So that’s why it was then hard-coded into the IBK node.”
“But then I added a bunch of check-boxes in there, which are still there,” continues Lambert. “If you open up the gizmo and then you go to the actual IBK node, there are additional check-boxes which you don’t get on the public version, which allow you to get access to the floating point alpha, and to turn off certain things which legalize the RGBA relationship.”
What Lambert also found as he investigated colored shapes of blue and green and grads of blue and green, and the varying up all of the background in an image (see the bottom of this article for a reproduction of some of the original grads and shapes Lambert experimented with), was that to get the best amount of detail from one image to another, a compositor did not need to think of the process as purely a layered extraction, but instead almost like an image transference. This comes back to the idea of an alpha value being more than one.
“In a compositing program,” Lambert explains, “the alpha is usually clamped between zero and one. But because it’s floating point, it can have negative values. Honestly, you have to scratch your brain as to think, ‘Well, what the heck does a minus value in an alpha mean, or even what an alpha value over one is?’ But I know from a keying perspective of taking one foreground of a background and putting that onto a different background, it passes through this alpha, this floating point alpha, which gives you additional detail.”
“This is why I always advocate seeing what the screen subtraction, along with bkg luminance and chroma checked on, gets you first, as this is using the floating point alpha internally and will sometimes give you more integrated detail in the edges rather than just premult’ing. It does mean having to structure your comp differently from a traditional separated alpha but I’ve always felt keying is a different mindset to comping CG layers, for example.”
Ultimately, Lambert worked on the algorithm and a paper for the IBK keyer and delivered this to Digital Domain in April 2005 after toying away on it since the end of 2004. “It was hard-coded in June 2005 and then it was released worldwide in Nuke 4.5 from D2 Software in December 2005. I basically presented DD with a sheet of paper outlining the algorithm, writing down the expression for the basic keying approach, and then a group of nodes, which is how I figured it out, and then turned that into the expression. Literally, the core algorithm is five nodes. Yes, it does have a couple of expression nodes in there, but it’s pretty simple.”
Lambert says, at first, artists at Digital Domain were only tinkering with it but that the IBK keyer became more central when he and a team of compositors used it extensively on Pirates of the Caribbean: At World’s End in 2006 (the film was released in 2007). “It caught a lot of traction there because we had some very challenging bluescreens and it just seemed to work well for that work.”
Notes on IBK (from its inventor)
A number of compositors befores & afters spoke to say that, these days, they tend to try a few different keyers in Nuke to see which produces the best outcome, with each compositor mentioning the the IBK keyer always produced robust and consistent results, including for uneven backings. Lambert mentions that he is constantly surprised with the way the IBK keyer gets used. For example, he says, Wētā FX have made IBK the core keying tool for all their movies but have expanded the toolset with innovative artist controls – exactly how Lambert wanted the tool to progress.
He’s also surprised at some of the discussion around IBK relating to spill suppression, edge mattes and clean plates.
“Well,” says Lambert, on this topic, “I just want to say that IBK provides no form of spill suppression whatsoever. There is nothing in IBK which will give spill suppression. What it does do, when you feel as if there is a color being suppressed, it’s simply the foreground color times the matte being subtracted from the foreground. There is no spill suppression in the classic sense of what a spill progression is because, again, I didn’t want to put that into the node as well. I wanted that to be separate because whatever facility you come across, they all have their own spill suppression nodes, for blue and for green and various ways of actually doing it. So, rather than adding more sliders, I wanted to keep that separate so people could do that and not embed it into IBK.”
Then there’s the notion of the ‘clean plate’ generation in the IBK workflow. Lambert advises that he never intended for that to be part of the IBK keying process. “You can see how it can be called a clean plate, but the idea was that you took from the original image and you expanded out the edges. What that meant is that if you had a really bad blue or greenscreen which was varying in light per frame, it would track with it, it would work with it. The thing is, I can’t ever see a world of, if you’ve got a flashlight up against a bluescreen, how you would make a clean plate of that, it just doesn’t happen.”
Lambert adds: “That was the intention, in a very simplistic way, to pull a hard matte, expand the edges out with a very simplistic compositing tool, which basically was the foundation of getting rid of wires in VFX. The idea was to expand those edges out, which then provided the color which you should be keying that edge out from. So, again, simplicity was my goal, so that people could expand it out.”
For Lambert, personally, the big breakthrough he recognizes IBK brought in terms of keying and compositing was “this idea that it wasn’t a single color which you could define your key with, but it could be on a per pixel basis allowing a multitude of compositing processes to be applied to the input image color. And that’s where the big difference was between doing a clean plate and doing it the IBK way.”
The visual effects supervisor, who most recently called on his very particular expertise in compositing to implement ‘sandscreens’ on Dune, is of course adamant that the IBK keyer–or any keyer–is just part of the arsenal of tools a compositor can use in crafting VFX shots.
“The biggest thing to try and get people to realize is that there isn’t one magic set-up or one magic button which will pull your perfect key. There just isn’t.”
BONUS section: original grads and shapes reproduced by Paul Lambert
Feature image courtesy Hugo Guerra.
