Using the Stencil Buffer in Unity Free

News concerning the recent release of Unity 4.6 was largely dominated by its new UI system.

However, closer inspection of the 4.6 release notes reveals some other interesting changes and improvements, including the announcement that “Stencil buffer is now available in Unity Free”.

Stencil buffers are not exactly new technology – they’ve been around for at least 10 years, and available in Unity Pro since Version 4.2. They can be used in somewhat similar circumstances to RenderTextures (still a Pro-only feature) to create a range of nice effects, so I thought I’d have a play…

The stencil buffer is a general purpose buffer that allows you to store an additional 8bit integer (i.e. a value from 0-255) for each pixel drawn to the screen. Just as shaders calculate RGB values to determine the colour of pixels on the screen, and z values for the depth of those pixels drawn to the depth buffer, they can also write an arbitrary value for each of those pixels to the stencil buffer. Those stencil values can then be queried and compared by subsequent shader passes to determine how pixels should be composited on the screen.

For example, adding the following tags to a shader will cause it to write the value "1" to the stencil buffer for each pixel that would be drawn to the screen:

    // Write the value 1 to the stencil buffer
    Stencil
    {
        Ref 1
        Comp Always
        Pass Replace
    }

With a few more tags, we can turn this shader into a mask whose only function is to write to the stencil buffer – i.e.  that does not draw anything to the screen itself, but sets values in the stencil mask to control what pixels are drawn to the screen by other shaders:

    Tags { "Queue" = "Geometry-1" }  // Write to the stencil buffer before drawing any geometry to the screen
    ColorMask 0 // Don't write to any colour channels
    ZWrite Off // Don't write to the Depth buffer

 

So, now this shader won’t have any visible output – all it does is write the value 1 to the stencil buffer for those pixels it would otherwise have rendered. We can now make use of this stencil buffer to mask the output of another shader, by adding the following lines:

    // Only render pixels whose value in the stencil buffer equals 1.
    Stencil {
      Ref 1
      Comp Equal
    }

 

The effect created by these two shaders is illustrated by the two models in the following screenshot:

• the picture frame contains an (invisible) quad which writes the value 1 to the stencil buffer for all pixels contained within the frame.
• the character standing behind the frame has a shader that tests the values in the stencil buffer and only renders those pixels that have a value of 1 – i.e. those pixels that are seen through the picture frame.

 

 

You can see the full list of stencil mask operators at http://docs.unity3d.com/Manual/SL-Stencil.html