VFX Firsts: What was the first digital wire-removal shot in a film?

An unlikely film for a first, but a huge leap forward in invisible VFX.

Today, digital wire removals are common visual effects tasks. Although there are several methods and software choices available, they still require meticulous attention to detail.

But how were wire removals done previously, and how were they done in the early days of digital VFX? How do you actually paint out the wire and leave what was ‘behind’ it intact? And how do you invisibly remove the wire without viewers realizing anything had been done to the scene at all?

These are things that we look at this week on VFX Firsts as we reveal the first digital wire removal done for a film. Joining me is visual effects digital artist Katie Morris, an expert in paint and roto with experience on scores of projects, including many at Industrial Light & Magic, and an fxphd trainer. You can follow Morris on Twitter and check out her open source roto and paint scripts on GitHub developed in conjunction with Magno Borgo.

There are major spoilers in the show notes below about which film may have been the ‘first’ to use a digital wire removal, but you can listen first to the podcast on Apple Podcasts or Spotify. It’s also embedded directly below. Also, here’s the RSS feed.

1. The ‘first’…

Willard Huyck’s Howard the Duck (1986) utilized digital wire removal by ILM for a scene in which the duck is pulled out of his apartment along with his easy chair. The effect was filmed with steel wires pulling the puppet and prop horizontally through multiple walls and sets.

A tool called Layerpaint was used to do the paint work for the wire removal at ILM for Howard the Duck. It was made by The Computer Graphics Division of Lucasfilm (before it was spun off to become a separate company Pixar), authored by Mark Leather in around 1985/86 as a 32-bit paint program on the Pixar Image Computer. It allowed artists to draw in raster graphics on transparent layers over film footage.

To aid in analyzing some of the history of digital wire removal at ILM, befores & afters contacted a number of ILMers about those early days. This includes George Joblove and Douglas Kay, who were co-founders and co-managers of ILM’s Computer Graphics Department. We also talked to Doug Smythe, a computer graphics supervisor at ILM still today.

First, Kay shared the following about the development of the tools behind the digital wire removal for Howard the Duck.

It was called Layerpaint. I believe Bruce Wallace, one of the members of our group at that time, was the principal “painter”. Layerpaint was a quite ingenious tool. It had two principal features that made wire removal possible and practical.

One was that it allowed an artist to work on a high resolution image while looking at a lower resolution monitor. I could probably pull out an old Cinefex to check my numbers but a standard full resolution image for us at that time from the laser film scanner, was 3456 by 2304. Our video displays were either 512 or 1024. Previously, an artist could only look at a section of a higher resolution image at a time but with Layerpaint they could work at any resolution, either the entire image or zoomed in, regardless of the display resolution, and the tool would always be updating the image at its full resolution.

The other of course was the layering. The ability to “paint through” from one layer to another allowed the artist to blend, using soft brushes, a clean plate to where the wires were on the hero take and remove them. It was still a laborious frame by frame process but reasonably straight forward.

A number of corresponding technologies made some of this digital wire removal work possible, such as ILM’s laser scanner, film output approach and the general hardware available back then. Kay continues:

The laser scanner/printer and the Pixar Image Computer were crucial. Also, a very clever idea to speed up and improve the quality of film recording by using 3 simultaneous red, green and blue lasers to write an image onto film rather than sequentially photographing a high resolution black and white monitor through 3 filters. It was sensitive device (utilizing a tiny mirror on the end of thin rod moving very quickly for the horizontal, while slowly moving it mechanically for the vertical) that often required adjustment but produced beautiful images.

The scanning was a bit more problematic. Since laser light is highly specular it would pick up all the detail of the grain in each frame of film. That was not a problem when looking at a still image but for a moving sequence, the grain would pop out creating too much noise. There was diffuser for the laser when in scanning mode to address this but ultimately we needed to come up with a different device for scanning on subsequent films. On Howard the Duck however, it was such a short shot, and there was so much going on in the shot to begin with (Howard crashing through walls on a sofa, with dust and debris everywhere), it was acceptable.

Although the film of course did not do all that well at the time, the wire removal shot allowed us to demonstrate the possibilities for digital technology in VFX. As was often the case, we got the opportunity because there was no other way to achieve the effect (the “wire” was a huge cable and none of the traditional techniques would have worked) and with success, we were able to build on it from there.

After Pixar was spun off and ILM’s Computer Graphics Department was created to take its place, development of Layerpaint was taken over by Jonathan Luskin, and then later by Doug Smythe.

The original version of Layerpaint saw each individual frame of a sequence having to be loaded one at a time by the artist, and each completed frame also saved out manually, which could be tedious and error-prone. In late 1987 or perhaps early 1988, Doug Smythe added a Sequencer module to Layerpaint, in which one could specify the base names and frame ranges for a complete sequence of frames for each layer, and buttons in the main UI to step forward or backward through the frames. Smythe ran down for befores & afters how this worked:

Each layer could also optionally be set to save any painted frames back to disk, so one could have a writable copy of the hero plate in the A layer, a safety copy of the same hero plate in the B layer, and the clean plate in the C layer, and then paint through from C to A to remove something, or from B to A if you messed up and had to restore something.

Images in each layer could be slid up, down, left or right independently and the frame numbers could be offset between sequences, so you could paint or “clone” from a source off to one side of a wire or from an earlier or later frame: this was handy when a wire crossed in front of a building or other complicated object: if a wire is over one part of a doorway or window on one frame, you looked for a frame where the doorway or window was not obscured and painted through from that.

2. Further digital wire removal breakthroughs on Back to the Future Part II

Digital wire removal was used for the hoverboard sequence in Back to the Future Part II for wires and also a pipe that supported the hoverboard. At this point, Doug Smythe wrote an interactive tool called “Wirem” that was used to indicate the top and bottom endpoints of any number of (straight) wires on keyframes, and interpolated the motion of the lines representing those wires.

Smythe explains:

The user could step through the playback of the lines over the background to check accuracy and lineup. Each wire also had a “thickness” value, which told the removal part of the process how many pixels wide each wire was. For thin wires, this was around 2-4, but for a few shots, where Marty McFly was standing on a hoverboard attached to the camera car by a thick black pole strong enough to support the actor’s weight, the “wire” thickness was closer to 30 or 40 pixels.

The automated wire removal process was devised by Les Dittert and coded by Dittert and Smythe. More again from Smythe:

It would read each input image in the sequence, then for each scanline (pixel row) in that image, it would determine which wires crossed that scanline and at what pixel column, then “smoosh” the color values from wirewidth/2 pixels on the left to wirewidth/2 pixels on the right to replace the wire pixels on that row. So if a wire was 4 pixels wide, and the color value was “40” 2 pixels to the left and “50” 2 pixels to the right, it would fill in values smoothly increasing from 40 to 50 over the 4 pixel width of the wire. Since computer image pixels actually consist of 3 separate values for the Red, Green and Blue color components, this interpolation and smooshing had to be repeated three times for each wire on each row. Extra math had to be done when two wires overlapped or crossed each other, and the program also had the option of adding a little bit of random color offset to the interpolated values to mimic the look of film grain.

Ultimately, several ILM players in this history of digital wire removal received a Scientific and Engineering Award (Plaque) in recognition of these efforts. The description of the award was: “To Mark Leather, Les Dittert, Douglas Smythe and George Joblove for the Concept and Development of the Digital Motion Picture Retouching System”, awarded February 26, 1994.

3. No Strings Attached

No Strings Attached, mentioned in the podcast, was an 8/16 bit paint program for SGI systems. It was an early wire removal tool written by Paul Miller, a prolific author of image processing tools in VFX, and a partner/lead software engineer at Silhouette FX. Here’s Paul’s notes which he shared about No Strings Attached:

It started out as an “automated” wire/scratch removal tool. We had a math guy working with us who was doing research on such stuff. But eventually we realized there is no such thing as “good automatic results” so I added a bunch of paint tools including a pretty novel clone tool. It turned into a general “fix it” tool and was bought by a lot of insurance companies in case the film was damaged (at 14K/seat!).

I dug up an old announcement for it here. I was pretty busy back then. I had just moved from Virginia to Wisconsin at the end of 1992 and had never used UNIX or seen an SGI before. I was hired by ASDG because of my “l33t Amiga skillz” but my boss Perry Kivolowitz pointed to their Indigo R3K and told me to “learn that” and by NAB I had written Elastic Reality, No Strings Attached, and an image conversion tool called Image Independence. A lot of 80-100 hour weeks, but it didn’t matter because I didn’t know anyone.

I also found this old “review” of our 1993 NAB booth:

“Across from Centaur stood another ASDG booth. Here, Perry & Gina
and crew had no Amigas but they were showing three new products for the Mac and SGI machines – “Elastic Reality – a Third Generation Morphing System” looked like Morph Plus with some pretty snazzy enhancements. I watched programmer Paul demo it on an SGI Indigo. And I gently fondled this lovely, purple, $20,000 machine as he did so.

Instead of vectors and edges it uses bezier curve drawing tools and splines. Very sexy stuff. “No Strings Attached” another new product is an Automatic Wire and Scratch Removal program using AI routines. Let’s say you’ve shot footage of a model airplane against a background suspended by wires or filament lines. This program will go thru the frames and intelligently remove the wires and lines and fill in the background seamlessly. Again, SGI only.

“Image Independence” looks like the ADPro image conversion stuff ported to Mac and SGI, with a difference – its graphical interface can be incorporated into other programs giving them the ability to read and write any supported image-file format.”

Paul Miller, again:

It’s kind of funny, in two years I’ll have been writing roto and paint software in some way for 30 years!

And see this fun look back at an ad for No Strings Attached, posted at Jeff Almasol’s @redefinery Twitter page (thanks, Jeff).

4. Digital wire removal resources

Boris FX’s Silhoutte

Scott Squires’ Commotion

Matador from Avid

Polygon Howard the Duck story

Howard the Duck CGM interview

Cinefex #43, covering Back to the Future sequels

Katie Morris’ fxphd course: Introduction to Silhouette and Paint, Part 1

Alvy Ray Smith’s Digital Paint Systems: An Anecdotal and Historical Overview