How RenderMan was used for the film’s coral reef.
If you’ve been following along with our retro RenderMan series – where we’ve been featuring previously published RenderMan materials – you’ll know that we recently dived into Pixar’s work behind Ratatouille. Now we’re jumping to Andrew Stanton’s Finding Nemo from 2003.
The shading of large, organic sets for computer graphics posed several challenges to any production, and for Nemo it was the shading of a large coral reef serving as a backdrop to a large section of film that had to be solved. Here’s Pixar’s notes on the shading side of the project, originally written by Chris Bernardi.
Overview
Working from pastels and sketches provided by the art department, Pixar was asked to create a completely invented environment yet still maintain a sense of realism. Through proper planning and the creative use of Pixarʼs PhotoRealistic RenderMan, the studio was able to tackle these challenges and provide a rich backdrop for the story. These notes look specifically at the process of shading the coral itself, from early shader design through to final optimizations.
Preproduction: What You Donʼt Know Can Hurt You
In January of 2001, Pixar began some of the initial pre-production tests for coral shading. Due to the many technical challenges facing the production, it was decided that they would build a miniature coral reef and take it all the way through the current pipeline to final film render. As a result of this test, the studio was able to discover several properties the coral shaders would need to possess.
Shader Flexibility
Shading changes for the coral need to be made late in the pipeline, possibly as late as shot lighting. New coral looks will need to be created on short notice.
Subsurface Scattering
None of the current illumination models were adequate to handle the subsurface scattering required for the coral. This look would need to be achieved with little setup time and minimal impact to render times.
Optimizations
With so much geometry in the set, optimizations will be required in both modeling and shading to achieve manageable render times.
Shader Flexibility
One Shader to Rule them All
From shading the corals in pre-production, Pixar discovered that all the corals that needed to be created had some similar shading properties. They also found that when they split the shader writing between too many people, there were problems maintaining a visual consistency on the set. This often led to corals, which had been modeled and shaded, going unused by the set dressing department. They simply did not work visually. This, coupled with the need for flexibility, led Pixar to believe that constructing a larger shader, with the ability to handle the majority of the corals, was going to be the best approach. The added benefit being that changes and optimizations could be handled from a single code base.
There has always been a continuum of approaches to shading: with the “one model/one shader” approach at one end of the spectrum, the “handful of shaders for everything” at the other. While Pixar often leans toward the former, the coral represented a situation where a more global shading approach was practical. This is often the case in natural, organic settings (e.g. it does not make sense to write a unique shader for every blade of grass in a field).
The studio’s initial thoughts were that they might end up with four or five shaders, but they would share a common collection of shading language functions. As it turned out, all coral in the film was shaded with a single shader. The same shader found its way into applications beyond the coral reef; such as rocks and the throat of Nigel, the pelican.
Scalar Fields
Pixar knew it would want to control the placement of the different shader layers along the length of the various corals. Since many of the models were subdivision meshes, lacking in coherent texture coordinates, artists chose to do this using scalar fields. A scalar field is simply a piece of information (in Pixar’s case, a float value) associated with a vertex (or CV), then smoothly interpolated. These are also known as primVars in shading systems like Slim. They are passed into the shader as a varying float. You can use this much like a st coordinate system, but with only a single value associated with each vertex. The following information was needed for each vertex: the distance (along the surface) to the base of the coral and the distance to the tip of the coral. Note that one of these is not necessarily the inverse of the other, given the branching structure of many corals.
For corals generated procedurally (such as the big plate corals), the scalar field assignment was built into the procedural code. Models built in Maya required more work. While Pixar had considered other techniques – such as unwrapping the models or painting the values with Artisan – they chose instead to develop a simple script in MEL. The technical director only needed to select the vertices that comprised the tips (or base) of the coral, then run the script.
The script would “walk” across the subdivision mesh out from those points, dropping increasing values into the scalar field. It would then go back to re-normalize the values into a 0-1 range. While the result was not a direct measure of the distance (a measure of the number of vertices traversed), the regularity of the meshes with which Pixar worked, made any differences inconsequential. The result is an additional set of values available in the shader, giving the studio an indication of its position along the length of the coral.
In addition to using scalar fields for the placement of shading layers, they were also used for shading tasks (such as suppressing polyp sizes color transitions). They ended up playing a critical role in the development of Pixar’s subsurface scattering look, as well.
Shading in Layers
Before getting into the nuts and bolts of writing a shader. It is usually a good idea to break apart the visual elements of what you are creating. Working closely with the Art Department, Pixar’s shading team were able to break the shading of the corals into three major elements, or layers:
Bone: the bony calcium carbonate “skeleton” of the coral
Velvet: the soft-looking main surface of the coral
Polyp: soft bulb structures that protrude out of the velvet layer
While these layers (and their names) are loosely based on the physical structure of real corals, they reflect the visual features Pixar wished to represent. This created a common language with the Art Department when coral features were being discussed. This was important since it was Pixar’s goal to match the Production Designerʼs vision, not to reproduce reality. With each layer existing in different physical locations across the coral, Pixar was able to treat them as independent shaders, each with its own illumination.
For both aesthetic reasons, and efficiency, transitions between the layers were usually abrupt. For example, Pixar never set the velvet layer to show through, all over, to the underlying bone layer. This would force the execution of all the code for each of the layers, including illumination. Instead, each layer was treated as a discreet, opaque layer of the coral; with some blending occurring in the transition areas. The transitions themselves often made use of the underlying patterns to make the transitions more interesting. See Appendix: More Interesting Transitions, below, for more details and code examples.
Patterns
The patterning of the layers (including coloration, bump/displacement and modulation of illumination parameters) was achieved through combinations of texture and procedural methods. 3D Painting, directly on the individual corals, was avoided due to the number of coral varieties required; as well as the complexity of setting up projections, or proper parameterization, for many of the complex branching corals. When Pixar needed texture coordinates, a modified version of a cubic projection was used instead, to accommodate the varied geometry.
All of the major patterns, available in the various layers, were set up in advance: with the ability to access over 30 different texture/procedural combinations. Some patterns were painted; some were generated procedurally in the shader; and others were initially written as RenderMan shaders, and then baked out into textures (for convenience and speed).
On layers such as the polyps, additional processing controls were added to allow maximum flexibility of the existing layers. This was particularly useful for displacement, where the results of the processing are directly seen as changes in geometry.
The combination of these three layers and a flexible system of pattern generation, gave us exceptional control over the look of the corals. Also providing a common code base and familiar set of controls, across all of the corals in the reef.
SubSurface Scattering: Well, Sort of….
Pixar knew, coming out of pre-production, that some form of subsurface scattering (referred to as “scattering” from now onwards) was going to be required. This is the effect when light enters the medium of an object, then is scattered instead of reflected. The reference, received from the Art Department, was to hold someone’s hand up to a light bulb and observe how light passed through the skin.
While much work has been done in this area, with several approaches underway at Pixar at the time, the coral shading had a unique set of constraints. The scattering look needed to incur little setup time on the part of the lighters. The effect could not impact render times significantly. This seemed to eliminate most approaches Pixar had seen so far. Instead, the team chose to approach the problem from a different angle. They knew they needed to look at this, not as a general sub-surface scattering solution, but a specific visual property that only needed to work for limited case corals. The resulting approach, while lacking in elegance, provides insight into alternative methodologies for approaching complex visual problems. The entire process was done over the course of several days, providing a fast and efficient way to achieve the specific look required.
The approach was unique for computer graphics illumination, but familiar to anyone with a background in sound synthesis. Illumination, in computer graphics, is usually based on an additive approach. Starting with a black surface, the various components of illumination (e.g. diffuse, specular etc.) are added up. The approach Pixar took with coral scattering was a subtractive approach (really a multiplicative, but that would be splitting hairs).
The idea was to assume every light has a 100% contribution to the entire surface of the coral, discarding the influence of view angles and surface normals. This is similar to ambient lighting techniques. For any light shining on the coral, the entire surface glows based on the intensity of the light. This was made into a simple illumination loop so Pixar could have access to shadowing.
At this point, a mock-up was assembled in Photoshop to explore useful terms for modulating this uniform glow. A variety of grayscale images were generated. These included: scalar fields, dot products (of the view angle and surface normal), some texture passes and layer masks. By working in a compositing environment, the studio was quickly able to determine which elements might prove useful to the final illumination procedure. In the end, the dot product of the view vector and the surface normal (a measure of how much the surface is facing the camera -referred to as normalDot from this moment on), a texture and a scalar field of the tip distance were all the elements initially required to give a satisfactory look. The resulting code resembled the following:
Cscatter = mix(Cinner, Couter, tipDistance*normalDot);
Cscater = mix(Cscatter, texColor, blurryTexture);
Cscatter *= scatterIllum(P);
Pixar used the normalDot as a naive measure of the distance the light would need to travel, if behind the object. While this is generally true for surfaces like spheres, it does not hold up so well for arbitrary geometry. By multiplying the scalar field for tip distance by the normalDot, the team was able to eliminate most of the “x-ray” effect, which often results from using values such as normalDot for illumination.
This was consistent with the direction from the Art Department, who wanted the scattering look more pronounced at the tips (or edges, in the case of the plate corals). Pixar used the result to mix an inner and outer color. The inner color tended to be more saturated, representing the color of the subsurface media. The outer color was less saturated and tended to take on more of the surface color. The team noticed that, if they calculated the normalDot prior to displacement of the polyps, they could achieve a more realistic look. This gave the polyps, on the silhouette edge of the object, a nice translucent feel. They then introduced a texture Pixar had blurred to represent varying densities in the subsurface media. Although this is really a hack, not providing any parallax or true volumetric effects, its use was subtle. It provided the extra level of detail needed.
The textures were generated in layers, with varying amounts of blur and transparency to fake a sense of depth. The final shader code offered controls over the influence of these various components into the final scattering. Pixar then factored the result into the new illumination loop the team had constructed.
The initial results were quite promising. The test results looked good when the light was behind an object, but not when the light was in front. While lighters could turn this illumination off on a per-light basis, Pixar decided to add a slight improvement to the illumination, to offer more control from within the shader. The team simply added a ramp to the effect; based on how much the light was pointing at the camera, versus away from the camera.
This not only solved the initial problem, but gave the overall illumination a much more natural feel. The position, and angle, of the light had a more realistic effect on the final look. The result being the creation of parameters; to control the tendency of light to scatter forward only, versus scattering in all directions. A series of animations, with the camera orbiting some typical coral, was used to balance the transition from diffuse to scattering illumination.
A quick glance at the approach reveals that shadows could cause significant problems to the illumination model. If the lighting has its greatest influence when directly behind an object, then a shadow (from that light) would essentially remove all the illumination created. Pixar had developed several schemes for handling this (such as reversing normals during shadow renders to cull the front faces). After several tests with some of the lighting crew, the team discovered – with judicious use of shadow bias, shadow blur and shadow density – that they preferred the look as it was.
Variation
Whenever you apply the same shader to a collection of geometry, you run the risk of all the models looking the same. This is particularly true if applying the shader to a limited set of models, as happened in the coral reef. Much of this can be eliminated by minor adjustments to the parameters of the shader, for each model. This can be a time-consuming task when there are thousands of models in a scene.
A simple technique was used to add variation to the corals and help break up these similarities. This approach begins by passing a unique integer to each of the models. This integer is then passed to a call to cellnoise() in the shader, resulting in each model having a unique float associated with it. This vector was then used to offset shader space, offset the colors (often in HSV space) and offset the overall size of the features. While the results can be subtle, the absence of these variations led to objectionable similarities among the coral.
Optimization
With the large amount of geometry Pixar planned on using in the reef, the studio knew that optimization was going to play an important role in managing rendering times. Aside from standard code optimizations, Pixar made extensive reuse of function calls and variables. Single channel textures of reasonable sizes (usually 1k) were used whenever possible, with color information introduced in the shader. Information flow was kept as float data, until it was needed as color information. Calls to noise (particularly fractal noise) were kept to a minimum and often reused in various aspects of the shading. It was often found that a texture was always being run through a series of processing functions, before it was used (smoothstep, pow etc.). When this occurred, a new texture was generated, with these functions baked in and the modifying functions turned off.
When working with large shaders which are going to be used for a variety of purposes, it is important to control what aspects of the shader are executed, for any given model. The shader was written in a modular fashion, bounded by conditional statements. The logic for these conditionals was based in the model. This allowed the model to execute the conditionals once per frame, shutting down large blocks of the shader (often entire layers) by passing a uniform parameter to the shader. For example, if the displacement parameters for the polyps were set to zero, then ignore all code (texture calls, modifiers, modifications of P) associated with that displacement.
The result of these optimizations gave Pixar an order of magnitude variation, in render time, for any given coral, based entirely on the complexity of the shading parameters. As a general practice, models were initially set with parameters that would execute quickly, only enhanced if required. While the team had considered a more automatic method, this approach did not require much time. This gave us the ability to make rendering speed decisions, based on all the aesthetic elements of a scene (including depth-of-field and fog).
While Pixar had considered using additional optimizations, such as level-of-detail and cards, they were pleased enough with the render times that additional work was not merited.
Conclusions
Large global shaders can, and should, play a specialized role in the shading pipeline of any large production. The success of the coral shader led to its application to a variety of surfaces, extending far beyond its original purpose. By focusing on its practicality and impact on lighting, no special setup was required to achieve the backlighting effects. The lighting lead, for the coral reef, commented: “I just point some lights at the coral and they look great.” This is music to the ears of anyone who has had to do last minute fixes to shaders, once they have entered into the hectic shot lighting phase of production.
Appendix: More Interesting Transitions
When you are creating a mask to blend, or mix, two elements in the shader, it almost always helps to break up the transition. For example:
uniform color Cyello = color (1, 1, 0);
uniform color Cblue = color (0, 0, .5);
float ramp = smoothstep (.25, .75, s);
color Cfinal = mix(Cyello, Cblue, ramp);
This is not very interesting. We would prefer to break this up in some fashion. While there are a number of ways one might go about this, here is a simple one:
uniform float distort = .25;
uniform color Cyello = color (1, 1, 0);
uniform color Cblue = color (0, 0, .5);
/*texture noise or any interesting pattern*/
float pattern = noise(s*20,t*20);
float ramp = smoothstep (.25, .75, s + (pattern -.5)*distort);
color Cfinal = mix(Cyello, Cblue, ramp);
Here we have offset the s component of the texture space in which we are calling the ramp by some interesting pattern.
Another approach involves changing the way you think about the smoothstep function. Rather than pass a high and low point for the smoothstep, you can write a little function to pass a center and a blur. This will function more like a step or threshold function, but with a variable blur component.
float blurstep (float center; float blur; float pattern){
return = smoothstep(center-blur/2,center+blur/2,pattern);
}
Using this new function, we can rewrite our transition in a more powerful way.
uniform float blur = .2;
uniform color Cyello = color (1,1,0);
uniform color Cblue = color (0,0,.5);
/*texture noise or any interesting pattern*/
float pattern = noise(s*20,t*20);
float ramp = blurstep(pattern, blur, smoothstep(.25, .75, s));
color Cfinal = mix(Cyello, Cblue, ramp);
The result is still a function that runs from 0 to 1 across s (well, basically), but we now have a far more interesting pattern and the ability to control the blur.
Donʼt forget that if you already have an interesting pattern variable that you are using somewhere else in your code, you might be able to reuse it here and pick up a lot of organic complexity at very little rendering cost.
