Deferred shading

Introduction

Today let's talk about how the graphics pipeline works. In general, a 3D scene is made up of primitives, textures, and shaders. Illumination is computed for every point in the scene, and then the point closest to the camera (the depth test) makes it to the screen, forming the final image. That's a rough description, but the details don't matter right now.

What matters is this: at the moment we compute illumination for a given point in the scene, there's no guarantee that point will actually contribute anything to the final frame — it might be occluded by another object and fail the depth test, and by then the graphics pipeline will have already spent resources computing its illumination.

In the example, only object A actually contributes to this pixel on screen (it occludes the other two), yet lighting gets computed for all three objects. This inefficiency grows as scene complexity increases. On top of that, it multiplies by the number of light sources, which quickly turns into a serious performance hit.

Deferred lighting

The solution to the problems described above is a technique called deferred shading (or deferred lighting).

The idea is that during the geometry pass, no lighting is computed at all. Instead, we write position, normal, color, and other attributes — basically everything needed to compute the final illumination — into a set of textures. Together these textures are called the g-buffer. After that, the final image is computed using the data from the g-buffer (the so-called lighting pass).

The g-buffer is a set of 2D textures. In other words, lighting is no longer computed in the 3D scene — it's computed on a 2D image. Every point for which lighting gets computed is a point of the final frame. That means lighting is only computed for points that passed the depth test and will actually show up in the frame; nothing is computed for the rest (the ones hidden behind other objects), which gives a substantial performance boost.

In effect, deferred shading can be thought of as a special kind of post-processing: the shader receives all the information needed to compute lighting as a set of 2D textures on its input, and produces the final frame as its output.

Pros and cons of the technique

Let's look at the advantages of deferred shading:

The g-buffer can be built in a single pass — the scene is drawn only once
Lighting is computed only for visible pixels
Support for a large number of light sources (and low cost when computing lighting for many of them)
Post-processing effects are easy to implement
Scenes with a variety of materials are easy to handle

Downsides of the technique (we'll talk about how to work around them at the end of the article):

Transparent and semi-transparent materials are hard to implement
Anti-aliasing is hard to implement

Preparing the data

The starting scene is a room with cubes and a 3D logo, lit with Unity's standard shader (which is essentially Phong shading).

Implementing deferred shading comes down to the following steps:

Set up an empty g-buffer and tell Unity that the output of the geometry pass (the result of the regular scene draw) should be written into the g-buffer
Write a material shader (a surface shader) that replaces the standard one — instead of computing lighting, its job is to fill the g-buffer
At this point we have a filled g-buffer and a completely black screen: first, because the scene is being drawn into the g-buffer, and second, because the material shader isn't computing any lighting — its only job is to fill the g-buffer
Write a deferred shader that takes the g-buffer data as input, computes lighting, and outputs the final image

Rendering to multiple textures in Unity (MRT)

The technique of rendering a scene into a texture instead of straight to the screen is called a render target. With deferred shading, a single pass needs to fill not just one but several textures at once. This technique is called multiple render target, or MRT for short.

Let's write a class that attaches to Unity's camera and switches it into MRT mode.

Initializing the variables:

cpp

// number of textures in our g-buffer
private const int MRT_COUNT = 2;

private Camera camera;

private RenderBuffer[] buffers = new RenderBuffer[MRT_COUNT];
private RenderTexture[] texes = new RenderTexture[MRT_COUNT];

Setting up the render

Next, we need to initialize the g-buffer and tell Unity's camera to render into the g-buffer instead of the screen.

cpp

void OnEnable()
{
    // build an empty buffer
    for (int i = 0; i < MRT_COUNT; i++) {
        texes[i] = new RenderTexture(Screen.width, Screen.height, 24, RenderTextureFormat.ARGB32);
        buffers[i] = texes[i].colorBuffer;
    }

    // tell the camera to render into the buffer
    // we also need depth written into the buffer (we'll need it later)
    camera = GetComponent<Camera>();
    camera.depthTextureMode = DepthTextureMode.Depth;
    camera.SetTargetBuffers(buffers, texes[0].depthBuffer);
}

Writing to the G-buffer (geometry pass)

At this point we have an empty g-buffer. The next task is to write a shader for a simple material that, instead of computing lighting, fills the g-buffer.

Vertex shader:

glsl

struct fragment_in
{
    float4 pos: POSITION;
    float2 uv : TEXCOORD1;
    float3 normal: TEXCOORD2;
};

fragment_in vert(vertex_data v)
{
    fragment_in o;

    o.pos = UnityObjectToClipPos(v.vertex);
    o.uv = v.uv;
    o.normal = UnityObjectToWorldNormal(v.normal);

    return o;
}

The vertex shader is pretty standard (position, normal, and UV get passed to the fragment shader), but the fragment shader is going to be a bit unusual. Let's start by defining the output structure:

glsl

struct fragment_out
{
    float4 color: COLOR0;
    float4 normal: COLOR1;
};

Note that the graphics pipeline figures out on its own that COLOR0 is the first texture in the G-buffer, COLOR1 is the second, and so on. Also, the depth texture isn't filled "by hand" — that happens automatically, since we already told Unity to do so in the script above.

Fragment shader:

glsl

fragment_out frag(fragment_in i): COLOR
{
    fragment_out o;

    float3 color = tex2D(_MainTex, i.uv * _MainTex_ST.xy);
    // if no texture is assigned, use the color from the _Color variable
    if (distance(color, 1.0) == 0) {
        color = _Color;
    }

    float3 N = normalize(i.normal);

    o.color = float4(color, 1.0);
    o.normal = float4(N * 0.5 + 0.5, 1.0);

    return o;
}

The components of a normal vector can be positive or negative (in the range from -1 to 1). By default, a texture can't store negative values, so the normal is multiplied by 0.5 (giving a range from -0.5 to 0.5), and then 0.5 is added, giving a range from 0 to 1. In the deferred shader we need to reverse this operation to recover the original normal value.

In its simplest form, the g-buffer holds just two textures: the point's color and its normal. Computing lighting also requires the point's position in world space, but there's no need to store that in the g-buffer. Given a point's position on screen and its value from the depth map (more on that later), we can reconstruct its world-space coordinates, so there's no reason to overload the g-buffer with redundant information.

If everything was done correctly, the g-buffer should now contain the following:

That wraps up the scene draw (geometry pass) — the g-buffer now holds all the information needed to compute lighting, and we can move on to the next step: the lighting pass.

Computing lighting

Reconstructing position from the depth map

Let's talk about how a 3D scene gets turned into a 2D image. As you probably know, a point in 3D space first goes from local space into model space, then into world coordinates, and is finally projected onto a 2D plane. Schematically, that pipeline looks like this:

With deferred shading, the developer faces the opposite problem: given a point's 2D coordinates on screen and its value from the depth map, we need to recover its coordinates in world space. That requires building the inverse of the MVP matrix. It needs to be built on the script side (not in the shader) — in our controller — since this matrix is the same for every point, and inverting a matrix is a fairly expensive operation.

cpp

// matrix that transforms from camera space (!) into world space
Matrix4x4 matrixCameraToWorld = camera.cameraToWorldMatrix;

// inverse projection matrix
Matrix4x4 matrixProjectionInverse = GL.GetGPUProjectionMatrix(camera.projectionMatrix, false).inverse;

// final result: matrix that transforms from screen coordinates into world space
Matrix4x4 matrixHClipToWorld = matrixCameraToWorld * matrixProjectionInverse;

This matrix needs to be passed into the deferred shader that computes the final lighting.

Passing data into the deferred shader

Let's extend the controller: create a material from the deferred shader, pass it the g-buffer plus everything else it needs, and apply the shader to a fresh (empty) frame that will hold the final lit scene once the deferred shader has run. So, first — creating the material and passing it the g-buffer:

cpp

private Material deffered_material;

if (!deffered_material) {
    var shader = Shader.Find("karonator/Deffered");

    if (shader != null) {
        deffered_material = new Material(shader);
        deffered_material.hideFlags = HideFlags.DontSave;
        
        for (int i = 0; i < texes.Length; i++) {
            // pass the g-buffer textures into the shader one by one,
            // named _Tex0, _Tex1, and so on
            deffered_material.SetTexture("_Tex" + i, texes[i]);
        }
    }
}

We also need to pass the light sources' positions and colors into the shader. There's a dedicated LightsManager class for that, but its internals aren't important right now. All you need to know is that this class can return the light sources' positions and colors as arrays of 3D vectors. Next, we pass that data into the shader:

cpp

void OnPreRender()
{
    LightsManager LM = GetComponent<LightsManager>();
    deffered_material.SetVectorArray("_LightsPositions", LM.lightsPositions());
    deffered_material.SetVectorArray("_LightsColors", LM.lightsColors());
}

The last step in getting the shader ready: passing it the screen-to-world transform matrix. Without this step it's impossible to compute a point's world-space coordinates, and consequently impossible to compute its lighting.

cpp

void OnRenderImage (RenderTexture source, RenderTexture destination)
{
    // recompute the screen-to-world transform matrix every frame
    Matrix4x4 matrixCameraToWorld = camera.cameraToWorldMatrix;
    Matrix4x4 matrixProjectionInverse = GL.GetGPUProjectionMatrix(camera.projectionMatrix, false).inverse;
    Matrix4x4 matrixHClipToWorld = matrixCameraToWorld * matrixProjectionInverse;

    // pass the matrix into the shader
    deffered_material.SetMatrix("clipToWorld", matrixHClipToWorld);

    // apply the shader to the frame
    Graphics.Blit(source, destination, deffered_material, 0);
}

All the necessary data has now been passed into the deferred shader, and we can move on to the final step: computing lighting.

Computing lighting (lighting pass)

Inputs:

cpp

uniform sampler2D _Tex0; // first texture from the g-buffer (color)
uniform sampler2D _Tex1; // second texture from the g-buffer (normals)
uniform sampler2D _CameraDepthTexture; // depth map

uniform float3 _LightsPositions[32]; // light source positions
uniform float3 _LightsColors[32]; // light source colors

uniform float4x4 clipToWorld; // transform matrix

A function to transform from screen space into world space:

cpp

float3 screenToWorld(float2 uv) {
    float depth = tex2D(_CameraDepthTexture, uv).x;

    // build the point in clip space using the value from the depth map
    float4 clipSpacePosition = float4(uv * 2.0 - 1.0, depth, 1.0);
    // transform it into world-space coordinates
    float4 worldPosition = mul(clipToWorld, clipSpacePosition);
    // after almost any matrix/coordinate operation like this,
    // don't forget to divide the result by w
    return worldPosition.xyz / worldPosition.w;
}

The lighting function:

cpp

float4 frag(v2f_img i): COLOR {
    float4 color = tex2D(_Tex0, i.uv);
    float4 raw_normal = tex2D(_Tex1, i.uv);
    float3 pos_world = screenToWorld(i.uv);

    float3 N = normalize(2 * (raw_normal.xyz - 0.5));

    float3 result = float3(0, 0, 0);
    for (int i = 0; i < 32; ++i)
    {
        float3 L = normalize(_LightsPositions[i] - pos_world);
        float dist = distance(_LightsPositions[i], pos_world);

        float contribution = 1.0 / (pow(dist, 2.0) + 0.0001);
        result += max(dot(L, N) * contribution, 0) * normalize(_LightsColors[i]);
    }

    return float4(result * color.xyz, 1.0);
}

The lighting function is fairly simple and is essentially the standard Lambert function, except each light source's contribution is multiplied by the contribution variable, which is inversely proportional to the squared distance from the point to the light source: the farther away the light, the less it contributes to the point's illumination.

Don't forget that normals are stored in the buffer in a packed form, so before using them you need to reverse that transformation to recover the original normal vector and compute lighting correctly.

Normal mapping

As a small bonus, we can add normal mapping to our deferred shading setup. We won't even need to touch the deferred shader — it's enough to "perturb" the normals right when they're written into the g-buffer, using data from a normal map, which means just slightly extending the surface shader.

In the vertex shader we need to build the normal, tangent, and binormal vectors and pass them to the fragment shader. With Unity this is quite simple: the engine computes the normal and tangent for us and passes them into the vertex shader, and the binormal is just their cross product:

Normal (blue), tangent (red), binormal (green)

Vertex shader:

cpp

o.normal = UnityObjectToWorldNormal(v.normal);
o.tangent = UnityObjectToWorldDir(v.tangent.xyz);
o.binormal = normalize(cross(o.normal, o.tangent)) * v.tangent.w;

Fragment shader:

We read the value from the normal map and multiply it by the TBN matrix.

cpp

float3 tangentNormal = UnpackNormal(tex2D(_NormalMap, i.uv * _MainTex_ST.xy));
float3x3 TBN = float3x3(normalize(i.tangent), normalize(i.binormal), normalize(i.normal));
TBN = transpose(TBN);

float3 N = normalize(mul(TBN, tangentNormal));

A detailed breakdown of how normal mapping works and what the TBN matrix is will most likely show up on this site later as its own article. For now, there's a link at the bottom of this article describing how the normal mapping algorithm works.

Technology limitations

To wrap up, let's briefly talk about the downsides of deferred shading and possible ways to work around the issues you're likely to run into when using this rendering technique in practice, in a real project.

Full-screen anti-aliasing (AntiAliasing):

Standard full-screen anti-aliasing (MSAA) doesn't work out of the box with deferred shading, but it can be implemented: you'd need to create the g-buffer textures at some multiple of the screen resolution, and do the anti-aliasing by sampling those textures in the deferred shader.

That said, this isn't really the fashionable way to do it these days (the approach above is computationally quite inefficient). Most modern anti-aliasing methods are based on detecting edges in the frame (essentially applying edge detection to the depth map). We've already covered a similar technique on this site in the article about toon shading, except there we were outlining edges, whereas here they get blurred instead, making them less sharp.

Semi-transparent objects:

Rendering semi-transparent objects with deferred shading isn't the most trivial task, since the whole scene gets drawn into a texture in a single pass, and to render a semi-transparent object we need to draw both the object itself and whatever is behind it.

One workable, if not the most efficient, way to solve this is to render semi-transparent objects the "regular" way (forward rendering) after the deferred pass. This approach requires supporting both the deferred and forward pipelines in code, which adds to the overall complexity of the renderer.

There's also an approach where opaque objects are drawn into the g-buffer first, followed by transparent ones (into separate texture layers), sorted by distance from the camera. This approach has its own downsides too: the g-buffer grows larger, and the algorithm needs more compute resources.

The specific way you combine transparency with deferred shading depends heavily on the particulars of your project.

Multiple materials in a frame:

Supporting different materials with deferred shading almost always means adding more textures to the g-buffer — ones that record which material each screen point belongs to and what that material's parameters are. The downsides of this approach are obvious: first, the g-buffer grows in both complexity and size, and second, rendering every material is still the job of a single shader (the deferred shader), which can end up quite complex once you have a lot of materials, hurting both its performance and how maintainable it is.

Links and files

I'd like to sincerely thank user refroqus from the gamedev.ru forum for the great 3D model of my logo. Thanks, buddy.

Computer graphicsshadingunitydeferred-shadingnormal-mapping