Creating Windowless Unity applications

I remember some while back there was a trend for applications to seem integrated with your Windows desktop – animated cats that would walk along your taskbar – things like that… What about a windowless Unity game that used a transparent background to reveal your regular desktop behind anything rendered by the camera?

You can do this using a combination of the OnRenderImage() event of a camera, some calls into the Windows native APIs, and a custom shader. Here’s how:

First, you’ll need to create a custom shader, as follows:

Shader "Custom/ChromakeyTransparent" {
	Properties {
		_MainTex ("Base (RGB)", 2D) = "white" {}
		_TransparentColourKey ("Transparent Colour Key", Color) = (0,0,0,1)
		_TransparencyTolerance ("Transparency Tolerance", Float) = 0.01 
	}
	SubShader {
		Pass {
			Tags { "RenderType" = "Opaque" }
			LOD 200
		
			CGPROGRAM

			#pragma vertex vert
			#pragma fragment frag
			#include "UnityCG.cginc"

			struct a2v
			{
				float4 pos : POSITION;
				float2 uv : TEXCOORD0;
			};

			struct v2f
			{
				float4 pos : SV_POSITION;
				float2 uv : TEXCOORD0;
			};

			v2f vert(a2v input)
			{
				v2f output;
				output.pos = mul (UNITY_MATRIX_MVP, input.pos);
				output.uv = input.uv;
				return output;
			}
		
			sampler2D _MainTex;
			float3 _TransparentColourKey;
			float _TransparencyTolerance;

			float4 frag(v2f input) : SV_Target
			{
				// What is the colour that *would* be rendered here?
				float4 colour = tex2D(_MainTex, input.uv);
			
			    // Calculate the different in each component from the chosen transparency colour
				float deltaR = abs(colour.r - _TransparentColourKey.r);
				float deltaG = abs(colour.g - _TransparentColourKey.g);
				float deltaB = abs(colour.b - _TransparentColourKey.b);

				// If colour is within tolerance, write a transparent pixel
				if (deltaR < _TransparencyTolerance && deltaG < _TransparencyTolerance && deltaB < _TransparencyTolerance)
				{
					return float4(0.0f, 0.0f, 0.0f, 0.0f);
				}

				// Otherwise, return the regular colour
				return colour;
			}
			ENDCG
		}
	}
}


This code should be fairly self-explanatory – it looks at an input texture and, if a pixel colour is within a certain tolerance of the chosen key colour, it is made transparent (you may be familiar with this technique as “chromakey”, or “green/blue screen” used in film special effects).

Create a material using this shader, and choose the key colour that you want to replace (and change the tolerance if necessary). Note that you don’t need to assign the main texture property – we’ll use the output of a camera to supply this texture, which is done in the next step…

Now, create the following C# script and attach to your main camera:

using System;
using System.Runtime.InteropServices;
using UnityEngine;

public class TransparentWindow : MonoBehaviour
{
    [SerializeField]
    private Material m_Material;

    private struct MARGINS
    {
        public int cxLeftWidth;
        public int cxRightWidth;
        public int cyTopHeight;
        public int cyBottomHeight;
    }

    // Define function signatures to import from Windows APIs

    [DllImport("user32.dll")]
    private static extern IntPtr GetActiveWindow();

    [DllImport("user32.dll")]
    private static extern int SetWindowLong(IntPtr hWnd, int nIndex, uint dwNewLong);
    
    [DllImport("Dwmapi.dll")]
    private static extern uint DwmExtendFrameIntoClientArea(IntPtr hWnd, ref MARGINS margins);


    // Definitions of window styles
    const int GWL_STYLE = -16;
    const uint WS_POPUP = 0x80000000;
    const uint WS_VISIBLE = 0x10000000;

    void Start()
    {
        #if !UNITY_EDITOR
        var margins = new MARGINS() { cxLeftWidth = -1 };

        // Get a handle to the window
        var hwnd = GetActiveWindow();

        // Set properties of the window
        // See: https://msdn.microsoft.com/en-us/library/windows/desktop/ms633591%28v=vs.85%29.aspx
        SetWindowLong(hwnd, GWL_STYLE, WS_POPUP | WS_VISIBLE);
        
        // Extend the window into the client area
        See: https://msdn.microsoft.com/en-us/library/windows/desktop/aa969512%28v=vs.85%29.aspx 
        DwmExtendFrameIntoClientArea(hwnd, ref margins);
        #endif
    }

    // Pass the output of the camera to the custom material
    // for chroma replacement
    void OnRenderImage(RenderTexture from, RenderTexture to)
    {
        Graphics.Blit(from, to, m_Material);
    }
}

This code uses InterOpServices to make some calls into the Windows native API that change the properties of the window in which Unity runs. It then uses the OnRenderImage() event to send the output of the camera to a rendertexture. Drag the material to which you assigned the custom transparency shader into the m_Material slot, so that our chromakey replacement works on the output of the camera.

Then, and this is important: change the background colour of the camera to match the _transparentColourKey property of the transparent material.

image

This can be any colour you want, but you might find it easiest to use, say, lime green (0,255,0), or lurid pink (255,0,255).

And then build and run your game (you could enable this in editor mode, by commenting the #if !UNITY_EDITOR condition above, but I really don’t recommend it!). This should work in either Unity 4.x (Pro, since it uses rendertextures), or Unity 5.x any version.

Here’s Ethan from Unity’s stealth demo walking across Notepad++ on my desktop…:

transparent

This entry was posted in Game Dev and tagged , , . Bookmark the permalink.

One Response to Creating Windowless Unity applications

  1. rbrundritt says:

    Sweet!!! Next up, making app windows explode when you click the x button.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s