Adding Shadows to a Unity Vertex/Fragment Shader in 7 Easy Steps

This was a question asked on the Unity Forums recently, so I thought I’d just write up the answer here.

Unity provides its own unique brand of “surface shaders”, which make dealing with lighting and shadows relatively simple. But there are still plenty of occasions in which you find yourself writing more traditional vert/frag CG shaders, and needing to deal with shadows in those too.

Suppose you had written a custom vertex/fragment CG shader, such as the following simple example:

Shader "Custom/SolidColor" {
    SubShader {
        Pass {
        
            CGPROGRAM

            #pragma vertex vert
            #pragma fragment frag
            #include "UnityCG.cginc"

            struct v2f {
                float4 pos : SV_POSITION;
            };


            v2f vert(appdata_base v) {
                v2f o;
                o.pos = mul (UNITY_MATRIX_MVP, v.vertex);
                return o;
            }

            fixed4 frag(v2f i) : COLOR {
                return fixed4(1.0,0.0,0.0,1.0);
            }

            ENDCG
        }
    }
    Fallback "VertexLit"
}

 

This shader simply outputs the colour red for all fragments, as shown on the plane in the following image:

image

 

Now what if you wanted to add shadows to that surface? Unity already creates a shadowmap for you from all objects set to cast shadows, and defines several macros that make it easier to sample that shadowmap at the appropriate point. So here’s the changes you need to make to a shader to make use of those built-in shadows:

Shader "Custom/SolidColor" {
    SubShader {
        Pass {
        
            // 1.) This will be the base forward rendering pass in which ambient, vertex, and
            // main directional light will be applied. Additional lights will need additional passes
            // using the "ForwardAdd" lightmode.
            // see: http://docs.unity3d.com/Manual/SL-PassTags.html
            Tags { "LightMode" = "ForwardBase" }
        
            CGPROGRAM

            #pragma vertex vert
            #pragma fragment frag
            #include "UnityCG.cginc"

            // 2.) This matches the "forward base" of the LightMode tag to ensure the shader compiles
            // properly for the forward bass pass. As with the LightMode tag, for any additional lights
            // this would be changed from _fwdbase to _fwdadd.
            #pragma multi_compile_fwdbase

            // 3.) Reference the Unity library that includes all the lighting shadow macros
            #include "AutoLight.cginc"


            struct v2f
            {
                float4 pos : SV_POSITION;
				
                // 4.) The LIGHTING_COORDS macro (defined in AutoLight.cginc) defines the parameters needed to sample 
                // the shadow map. The (0,1) specifies which unused TEXCOORD semantics to hold the sampled values - 
                // As I'm not using any texcoords in this shader, I can use TEXCOORD0 and TEXCOORD1 for the shadow 
                // sampling. If I was already using TEXCOORD for UV coordinates, say, I could specify
                // LIGHTING_COORDS(1,2) instead to use TEXCOORD1 and TEXCOORD2.
                LIGHTING_COORDS(0,1)
            };


            v2f vert(appdata_base v) {
                v2f o;
                o.pos = mul (UNITY_MATRIX_MVP, v.vertex);
                
                // 5.) The TRANSFER_VERTEX_TO_FRAGMENT macro populates the chosen LIGHTING_COORDS in the v2f structure
                // with appropriate values to sample from the shadow/lighting map
                TRANSFER_VERTEX_TO_FRAGMENT(o);
                
                return o;
            }

            fixed4 frag(v2f i) : COLOR {
            
                // 6.) The LIGHT_ATTENUATION samples the shadowmap (using the coordinates calculated by TRANSFER_VERTEX_TO_FRAGMENT
                // and stored in the structure defined by LIGHTING_COORDS), and returns the value as a float.
                float attenuation = LIGHT_ATTENUATION(i);
                return fixed4(1.0,0.0,0.0,1.0) * attenuation;
            }

            ENDCG
        }
    }
    
    // 7.) To receive or cast a shadow, shaders must implement the appropriate "Shadow Collector" or "Shadow Caster" pass.
    // Although we haven't explicitly done so in this shader, if these passes are missing they will be read from a fallback
    // shader instead, so specify one here to import the collector/caster passes used in that fallback.
    Fallback "VertexLit"
}

 

And that’s it!

image

Posted in Game Dev | Tagged , , , | Leave a comment

Using the Stencil Buffer in Unity Free

News concerning the recent release of Unity 4.6 was largely dominated by its new UI system.

https://i0.wp.com/badgerheadgames.com/wp-content/uploads/2014/11/starcitizenhud.jpg

However, closer inspection of the 4.6 release notes reveals some other interesting changes and improvements, including the announcement that “Stencil buffer is now available in Unity Free”.

Stencil buffers are not exactly new technology – they’ve been around for at least 10 years, and available in Unity Pro since Version 4.2. They can be used in somewhat similar circumstances to RenderTextures (still a Pro-only feature) to create a range of nice effects, so I thought I’d have a play…

The stencil buffer is a general purpose buffer that allows you to store an additional 8bit integer (i.e. a value from 0-255) for each pixel drawn to the screen. Just as shaders calculate RGB values to determine the colour of pixels on the screen, and z values for the depth of those pixels drawn to the depth buffer, they can also write an arbitrary value for each of those pixels to the stencil buffer. Those stencil values can then be queried and compared by subsequent shader passes to determine how pixels should be composited on the screen.

For example, adding the following tags to a shader will cause it to write the value "1" to the stencil buffer for each pixel that would be drawn to the screen:

    // Write the value 1 to the stencil buffer
    Stencil
    {
        Ref 1
        Comp Always
        Pass Replace
    }


With a few more tags, we can turn this shader into a mask whose only function is to write to the stencil buffer – i.e.  that does not draw anything to the screen itself, but sets values in the stencil mask to control what pixels are drawn to the screen by other shaders:

    Tags { "Queue" = "Geometry-1" }  // Write to the stencil buffer before drawing any geometry to the screen
    ColorMask 0 // Don't write to any colour channels
    ZWrite Off // Don't write to the Depth buffer

 

So, now this shader won’t have any visible output – all it does is write the value 1 to the stencil buffer for those pixels it would otherwise have rendered. We can now make use of this stencil buffer to mask the output of another shader, by adding the following lines:

    // Only render pixels whose value in the stencil buffer equals 1.
    Stencil {
      Ref 1
      Comp Equal
    }

 

The effect created by these two shaders is illustrated by the two models in the following screenshot:

  • the picture frame contains an (invisible) quad which writes the value 1 to the stencil buffer for all pixels contained within the frame.
  • the character standing behind the frame has a shader that tests the values in the stencil buffer and only renders those pixels that have a value of 1 – i.e. those pixels that are seen through the picture frame.

 

stencil

 

You can see the full list of stencil mask operators at http://docs.unity3d.com/Manual/SL-Stencil.html

Posted in Game Dev | Tagged , , | Leave a comment

Optical / Spatial / Dimensional Trickery in Games

I’m currently considering a game design that makes use of various optical and logical tricks to fool the player as to how objects in the game are spatially-related – and thought I’d start by doing a bit of a review of what games are already out there that use similar techniques:

 

Sphinx Adventure

Released in 1982, this is one of the first computer games I can ever remember playing. Since the game was entirely text-based, there were no graphical tricks to deceive the player. However, that didn’t mean that you couldn’t be deceived and disoriented in other ways. The “trick” employed in this case was not a graphical one, but a logical one about the spatial connection between locations in the game.

This was back in the days before GameFAQs or games magazines, so players had to hand-draw maps of the game locations in order to find their way around. And, with only a sentence or two description for each room, it took a long time to figure out that rooms were sometimes connected to each other in impossible ways, creating infinite loops as demonstrated by the “iron passages” in this map extract…

https://i2.wp.com/www.stairwaytohell.com/gamehelp/MAP-SphinxAdventure-CP-3.jpg

 

The Legend of Zelda

The Lost Woods are a recurring theme that have appeared in numerous titles in the Legend of Zelda series. Here’s a screenshot of the original NES incarnation:

File:LoZlostwoods.png

It’s a “room” with several exits. Successful traversal through the woods required you to know the correct pattern of exits to follow through the rooms. Taking a correct choice led you gradually through the rooms of the forest. But taking an incorrect choice led you immediately back to the start room. However, since each room looked identical, there were no visual cues to suggest this was what was happening, and players were often left disorientated by the fact that simple logical truths proved incorrect (i.e. starting from a room, going North once and then South once did not put you back where you started).

 

EchoChrome

Echochrome was released on the Playstation over 6 years ago, although it feels more like a technical demo than a full released game. The technology which it demonstrates is the “Object locative environment coordinate system”, in which the relationship between objects in the virtual environment are not only determined by their 3d coordinates, but on the position from which they are viewed. Essentially if, from a certain point-of-view, two objects are made to look like they touch, then they do.

The levels were sparse and clearly M.C.Escher-inspired, meaning the entire focus of the game was concentrated on the gameplay mechanic of rotating the camera to create, and destroy, links between apparently separate platforms.

image

The exact same gameplay mechanic appear to have recently been re-used (in slightly more colourful form) in the game “Miika”:

image image
echochrome Miika

 

Monument Valley

Monument valley also appeared to take some inspiration from Echochrome, but developed the ideas further to include not only movement of the camera, but movement and rotation of parts of the level to create surfaces on which the player could walk (they also added a lovely art-style, making this feel much more of a developed game than Echochrome was). It still makes heavy use of the basic gameplay mechanic of making things appear connected in order to make them actually connected.

https://i0.wp.com/www.monumentvalleygame.com/img/screenshot3.jpg

 

Mystic Mine

Mystic Mine takes just one specific element of MC Escher’s iconic isometric style and turns it into a fun puzzle game. The mine cart will only ever go “down” slopes but, due to the optical illusion employed, it is always possible to reach every part of every level. It’s a neat, compact game, and you can even get the source code on GitHub.

image

 

Perspective

Most people are intuitively familiar with the concepts of a “2D game” and a “3D game”. “Perspective” challenges the boundaries between these genres by allowing the player to move around a 3D world in order to create a 2D world based on the view of world objects from the current camera position. Unlike echochrome, Monument Valley, Super Paper Mario etc., interaction is not dimensionally-restricted  – you can translate and rotate the camera in all three axes of the “3d” world to create an infinite variety of corresponding “2d” interpretations. It’s hard to describe and it’s surprisingly disorientating to play, but technically it’s very impressive to consider the level design process that must have gone into this:

http://www.seewithperspective.com/

 

Super Paper Mario

Super Paper Mario is hard to categorise as either a 2D game or a 3D game – rather, it offers the player the opportunity to switch between two different, but consistent, 2D perspectives of a 3D world. Like either looking at a cube from exactly front-on, or exactly side-on. The “paper”-thin theme allowed level elements and characters to have a physical presence in only two dimensions, meaning that obstacles in one view could easily be navigated by switching to the other view.

 

Fez

Fez creator Phil Fish says that Super Paper Mario is a terrible game, and that Fez is nothing like it. However, it’s hard not to see the similarity – with Fez similarly offering players different 2D perspectives of a 3D world:

 

Crush

I have to confess of never having heard of the game until I started writing this post, but apparently “Crush” on the PSP also uses a similar mechanic to Fez and Super Paper Mario – “crushing” 3D space into two dimensions:

File:Crush 3d.jpg File:Crush 2d.jpg

 

Tale of Scale

Tale of Scale” was created for the Ludum Dare 25 gamejam. Despite not really fitting the theme of “You are the Villain”, it featured a novel gameplay mechanic (and particularly impressive considering the 48hrs in which the game was developed) based around the apparent scale of in-game objects and forced perspective.

It’s somewhat reminiscent of the fantastic Father Ted episode in which Ted tries to teach Father Dougal about perspective:

Ted: [holding up a toy cow] All right, one more time. These… are small. The ones out there… are far away. Small. Far away

Dougal: [shakes his head in bewilderment]

The trick is that Tale of Scale dynamically adjusts the scale of objects such that objects that are small because they are “far away” can become actually small, and therefore can be picked up between the fingers.

https://i1.wp.com/ludumdare.com/compo/wp-content/compo2/201083/3663-shot3.jpg

 

It’s a technique familiar to anyone who’s ever taken holiday snaps of “pushing the Tower of Pisa back straight”, or “squeezing someone’s head between their fingers”, but to my knowledge it’s the first time it’s been used in a game as a puzzle mechanic.

The required code is actually surprisingly simple – here’s an implementation in Unityscript that will keep an object the same apparent size, whatever it’s true distance from the camera:

 

#pragma strict
private var startScale : Vector3;
private var startDist : float;
private var lastDist : float;

function Start() {
  startScale = transform.localScale;
  startDist = transform.position.z - Camera.main.transform.position.z;
  lastDist = startDist;
}

function Update() {
  var dist = transform.position.z - Camera.main.transform.position.z;
  if (dist != lastDist) {
    transform.localScale = startScale / startDist * dist;
    lastDist = dist;
  }
}

 

Museum of Simulation Technology

MoST uses very similar game mechanics based around forced perspective as introduced in Tale of Scale, and also includes familiar world landmarks as objects in the game to accentuate the player’s preconception of what should be large or small.

 

BluePrint 3D

Not really a “trick” here, but I thought I’d include it anyway. An object has been smashed into pieces which, when viewed from a certain angle, will reconstruct the appearance of the original object. Rotate the camera to find the correct viewpoint and the object will be revealed. It’s a simple mechanic but nicely implemented:

image

The same mechanic is also used in “Starlight”, and, IIRC, some elements of “The Room” series.

image

 

IIRC, there’s also a game where, when the completed object is seen to be constructed correctly, it becomes real in the 3D world, but I can’t remember what the game is right now.

Posted in Game Dev | Tagged , , , | Leave a comment

Previewing FBX Animations in Quicktime Player

I’ve been building up a library of FBX animations from a combination of MoCap stuff I’ve found around the internet and my own lovingly hand-crafted (though rather stilted – I’m not a natural animator!) clips.

I now have a folder of, for example, about 10 different walk cycles, but I’ve yet to find a good way of browsing those animations – a kind of video thumbnail preview integrated into Windows explorer would be ideal but, if that exists, I haven’t found it yet. In the meantime, one thing I have just found is this add-on that let’s you preview FBX animations in QuickTime player.

It’s a little clunky in places (including, for some reason, only allowing you to have a single FBX file loaded at a time, even after having started multiple instances of QuickTime player?)  but it’s certainly made it easier to browse through my animation library than importing each clip into Blender as I was doing previously. Range of features and shortcuts shown in the screenshot below.

QT_FBX

Posted in Game Dev | Tagged , , , | Leave a comment

Spacescape

Everyone likes a good space game, right? Then you’ll need some way of creating spacey backdrops. You could go for an 8bit parallax pixel starfield, or, at the other extreme, you might use authentic NASA imagery of the night sky.

Or, if you want high-res original, believable space backdrops, I highly recommend Spacescape on SourceForge. It’s a free application, licensed under MIT, and can generate beautiful parameterised space backgrounds.

spacescape

Like this:

space_up

What’s more, it doesn’t just generate single images – you can create a set of 6 cubemaps ready to import straight into a Unity skybox and get a full 360 space panorama. Fantastic asset for any developers looking for resources for a space-themed game.

Posted in Game Dev | Tagged , , | Leave a comment

Grabbing and Converting Adobe Streaming Video Fragments

I was recently interviewed by Mustard TV for a feature on the Norwich Sound & Vision festival, in which I’ve been involved. The feature was broadcast on Freeview cable TV but I can’t pick up the channel where I live, so I wanted to grab a copy off the website instead. They deliver videos using Adobe’s adaptive streaming protocol, which results in videos being sent in many fragments, of potentially different bitrates.

Here’s a little guide of what I had to do to save, join, and then convert the fragments into a single .MP4 video file.

 

1.) First, I went to the page in which the video was embedded and played the video the whole way through.

image

IMPORTANT NOTE:  Since Adobe’s streaming media server chunks videos into many different fragments, you need to watch the entire video the whole way through to make sure all the fragments are requested and downloaded.

 

2.) Once the video has ended, I used the media tab of the FlashGot FireFox extension to save all the video fragments to a local folder.

image

 

IMPORTANT NOTE: Due to the nature of adaptive bitrate streaming, you’ll see that the resolution of some of the fragments above varies depending on what my internet speed was like at the time. (Fragments 4 and 5 are only 600×640 resolution rather than 1200×1080 – I must’ve had some sort of spike at that point). To join a single video, you’ll want to ensure that you end up with a complete set of fragments at the same resolution. For fragments 4 and 5 above, I simply replayed that part of the video and it re-requested the file at a better resolution – you could also probably copy the URL template and specify the resolution of the fragment you want.

 

3.) To join the fragments together, I then used a PHP script which I found here. I’m reproducing it below for convenience.

To use the script, start a new command prompt and call it as follows, with the full filename omitting the fragment number:

php AdobeHDS.php --fragments The_Music_Mash_Mustard_TV_video_1200_1080Seg1_Frag

 

image

 

AdobeHDS.php

 1)
            {
              $paramSwitch = false;
              for ($i = 1; $i error("[param] expected after '$paramSwitch' switch (" . self::$ACCEPTED[1][$paramSwitch] . ')');
                  else if (!$paramSwitch and !$isSwitch)
                    {
                      if ($handleUnknown)
                          $this->params['unknown'][] = $arg;
                      else
                          $this->error("'$arg' is an invalid option, use --help to display valid switches.");
                    }
                  else if (!$paramSwitch and $isSwitch)
                    {
                      if (isset($this->params[$arg]))
                          $this->error("'$arg' switch can't occur more than once");

                      $this->params[$arg] = true;
                      if (isset(self::$ACCEPTED[1][$arg]))
                          $paramSwitch = $arg;
                      else if (!isset(self::$ACCEPTED[0][$arg]))
                          $this->error("there's no '$arg' switch, use --help to display all switches.");
                    }
                  else if ($paramSwitch and !$isSwitch)
                    {
                      $this->params[$paramSwitch] = $arg;
                      $paramSwitch                = false;
                    }
                }
            }

          // Final check
          foreach ($this->params as $k => $v)
              if (isset(self::$ACCEPTED[1][$k]) and $v === true)
                  $this->error("[param] expected after '$k' switch (" . self::$ACCEPTED[1][$k] . ')');
        }

      function displayHelp()
        {
          LogInfo("You can use script with following switches:\n");
          foreach (self::$ACCEPTED[0] as $key => $value)
              LogInfo(sprintf(" --%-17s %s", $key, $value));
          foreach (self::$ACCEPTED[1] as $key => $value)
              LogInfo(sprintf(" --%-9s%-8s %s", $key, " [param]", $value));
        }

      function error($msg)
        {
          LogError($msg);
        }

      function getParam($name)
        {
          if (isset($this->params[$name]))
              return $this->params[$name];
          else
              return false;
        }
    }

  class cURL
    {
      var $headers, $user_agent, $compression, $cookie_file;
      var $active, $cert_check, $fragProxy, $maxSpeed, $proxy, $response;
      var $mh, $ch, $mrc;
      static $ref = 0;

      function cURL($cookies = true, $cookie = 'Cookies.txt', $compression = 'gzip', $proxy = '')
        {
          $this->headers     = $this->headers();
          $this->user_agent  = 'Mozilla/5.0 (Windows NT 5.1; rv:26.0) Gecko/20100101 Firefox/26.0';
          $this->compression = $compression;
          $this->cookies     = $cookies;
          if ($this->cookies == true)
              $this->cookie($cookie);
          $this->cert_check = false;
          $this->fragProxy  = false;
          $this->maxSpeed   = 0;
          $this->proxy      = $proxy;
          self::$ref++;
        }

      function __destruct()
        {
          $this->stopDownloads();
          if ((self::$ref cookie_file))
              unlink($this->cookie_file);
          self::$ref--;
        }

      function headers()
        {
          $headers[] = 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8';
          $headers[] = 'Connection: Keep-Alive';
          return $headers;
        }

      function cookie($cookie_file)
        {
          if (file_exists($cookie_file))
              $this->cookie_file = $cookie_file;
          else
            {
              $file = fopen($cookie_file, 'w') or $this->error('The cookie file could not be opened. Make sure this directory has the correct permissions.');
              $this->cookie_file = $cookie_file;
              fclose($file);
            }
        }

      function get($url)
        {
          $process = curl_init($url);
          $options = array(
              CURLOPT_HTTPHEADER => $this->headers,
              CURLOPT_HEADER => 0,
              CURLOPT_USERAGENT => $this->user_agent,
              CURLOPT_ENCODING => $this->compression,
              CURLOPT_TIMEOUT => 30,
              CURLOPT_RETURNTRANSFER => 1,
              CURLOPT_FOLLOWLOCATION => 1
          );
          curl_setopt_array($process, $options);
          if (!$this->cert_check)
              curl_setopt($process, CURLOPT_SSL_VERIFYPEER, false);
          if ($this->cookies == true)
            {
              curl_setopt($process, CURLOPT_COOKIEFILE, $this->cookie_file);
              curl_setopt($process, CURLOPT_COOKIEJAR, $this->cookie_file);
            }
          if ($this->proxy)
              $this->setProxy($process, $this->proxy);
          $this->response = curl_exec($process);
          if ($this->response !== false)
              $status = curl_getinfo($process, CURLINFO_HTTP_CODE);
          curl_close($process);
          if (isset($status))
              return $status;
          else
              return false;
        }

      function post($url, $data)
        {
          $process   = curl_init($url);
          $headers   = $this->headers;
          $headers[] = 'Content-Type: application/x-www-form-urlencoded;charset=UTF-8';
          $options   = array(
              CURLOPT_HTTPHEADER => $headers,
              CURLOPT_HEADER => 1,
              CURLOPT_USERAGENT => $this->user_agent,
              CURLOPT_ENCODING => $this->compression,
              CURLOPT_TIMEOUT => 30,
              CURLOPT_RETURNTRANSFER => 1,
              CURLOPT_FOLLOWLOCATION => 1,
              CURLOPT_POST => 1,
              CURLOPT_POSTFIELDS => $data
          );
          curl_setopt_array($process, $options);
          if (!$this->cert_check)
              curl_setopt($process, CURLOPT_SSL_VERIFYPEER, false);
          if ($this->cookies == true)
            {
              curl_setopt($process, CURLOPT_COOKIEFILE, $this->cookie_file);
              curl_setopt($process, CURLOPT_COOKIEJAR, $this->cookie_file);
            }
          if ($this->proxy)
              $this->setProxy($process, $this->proxy);
          $return = curl_exec($process);
          curl_close($process);
          return $return;
        }

      function setProxy(&$process, $proxy)
        {
          $type      = "";
          $separator = strpos($proxy, "://");
          if ($separator !== false)
            {
              $type  = strtolower(substr($proxy, 0, $separator));
              $proxy = substr($proxy, $separator + 3);
            }
          switch ($type)
          {
              case "socks4":
                  $type = CURLPROXY_SOCKS4;
                  break;
              case "socks5":
                  $type = CURLPROXY_SOCKS5;
                  break;
              default:
                  $type = CURLPROXY_HTTP;
          }
          curl_setopt($process, CURLOPT_PROXY, $proxy);
          curl_setopt($process, CURLOPT_PROXYTYPE, $type);
        }

      function addDownload($url, $id)
        {
          if (!isset($this->mh))
              $this->mh = curl_multi_init();
          if (isset($this->ch[$id]))
              return false;
          $download =& $this->ch[$id];
          $download['id']  = $id;
          $download['url'] = $url;
          $download['ch']  = curl_init($url);
          $options         = array(
              CURLOPT_HTTPHEADER => $this->headers,
              CURLOPT_HEADER => 0,
              CURLOPT_USERAGENT => $this->user_agent,
              CURLOPT_ENCODING => $this->compression,
              CURLOPT_LOW_SPEED_LIMIT => 1024,
              CURLOPT_LOW_SPEED_TIME => 10,
              CURLOPT_BINARYTRANSFER => 1,
              CURLOPT_RETURNTRANSFER => 1,
              CURLOPT_FOLLOWLOCATION => 1
          );
          curl_setopt_array($download['ch'], $options);
          if (!$this->cert_check)
              curl_setopt($download['ch'], CURLOPT_SSL_VERIFYPEER, false);
          if ($this->cookies == true)
            {
              curl_setopt($download['ch'], CURLOPT_COOKIEFILE, $this->cookie_file);
              curl_setopt($download['ch'], CURLOPT_COOKIEJAR, $this->cookie_file);
            }
          if ($this->fragProxy and $this->proxy)
              $this->setProxy($download['ch'], $this->proxy);
          if ($this->maxSpeed > 0)
              curl_setopt($process, CURLOPT_MAX_RECV_SPEED_LARGE, $this->maxSpeed);
          curl_multi_add_handle($this->mh, $download['ch']);
          do
            {
              $this->mrc = curl_multi_exec($this->mh, $this->active);
            } while ($this->mrc == CURLM_CALL_MULTI_PERFORM);
          return true;
        }

      function checkDownloads()
        {
          if (isset($this->mh))
            {
              curl_multi_select($this->mh);
              $this->mrc = curl_multi_exec($this->mh, $this->active);
              if ($this->mrc != CURLM_OK)
                  return false;
              while ($info = curl_multi_info_read($this->mh))
                {
                  foreach ($this->ch as $download)
                      if ($download['ch'] == $info['handle'])
                          break;
                  $array['id']  = $download['id'];
                  $array['url'] = $download['url'];
                  $info         = curl_getinfo($download['ch']);
                  if ($info['http_code'] == 0)
                    {
                      /* if curl fails due to network connectivity issues or some other reason it's *
                       * better to add some delay before next try to avoid busy loop.               */
                      LogDebug("Fragment " . $download['id'] . ": " . curl_error($download['ch']));
                      usleep(1000000);
                      $array['status']   = false;
                      $array['response'] = "";
                    }
                  else if ($info['http_code'] == 200)
                    {
                      if ($info['size_download'] >= $info['download_content_length'])
                        {
                          $array['status']   = $info['http_code'];
                          $array['response'] = curl_multi_getcontent($download['ch']);
                        }
                      else
                        {
                          $array['status']   = false;
                          $array['response'] = "";
                        }
                    }
                  else
                    {
                      $array['status']   = $info['http_code'];
                      $array['response'] = curl_multi_getcontent($download['ch']);
                    }
                  $downloads[] = $array;
                  curl_multi_remove_handle($this->mh, $download['ch']);
                  curl_close($download['ch']);
                  unset($this->ch[$download['id']]);
                }
              if (isset($downloads) and (count($downloads) > 0))
                  return $downloads;
            }
          return false;
        }

      function stopDownloads()
        {
          if (isset($this->mh))
            {
              if (isset($this->ch))
                {
                  foreach ($this->ch as $download)
                    {
                      curl_multi_remove_handle($this->mh, $download['ch']);
                      curl_close($download['ch']);
                    }
                  unset($this->ch);
                }
              curl_multi_close($this->mh);
              unset($this->mh);
            }
        }

      function error($error)
        {
          LogError("cURL Error : $error");
        }
    }

  class F4F
    {
      var $audio, $auth, $baseFilename, $baseTS, $bootstrapUrl, $baseUrl, $debug, $duration, $fileCount, $filesize, $fixWindow;
      var $format, $live, $media, $metadata, $outDir, $outFile, $parallel, $play, $processed, $quality, $rename, $video;
      var $prevTagSize, $tagHeaderLen;
      var $segTable, $fragTable, $segNum, $fragNum, $frags, $fragCount, $lastFrag, $fragUrl, $discontinuity;
      var $prevAudioTS, $prevVideoTS, $pAudioTagLen, $pVideoTagLen, $pAudioTagPos, $pVideoTagPos;
      var $prevAVC_Header, $prevAAC_Header, $AVC_HeaderWritten, $AAC_HeaderWritten;

      function __construct()
        {
          $this->auth          = "";
          $this->baseFilename  = "";
          $this->bootstrapUrl  = "";
          $this->debug         = false;
          $this->duration      = 0;
          $this->fileCount     = 1;
          $this->fixWindow     = 1000;
          $this->format        = "";
          $this->live          = false;
          $this->metadata      = true;
          $this->outDir        = "";
          $this->outFile       = "";
          $this->parallel      = 8;
          $this->play          = false;
          $this->processed     = false;
          $this->quality       = "high";
          $this->rename        = false;
          $this->segTable      = array();
          $this->fragTable     = array();
          $this->segStart      = false;
          $this->fragStart     = false;
          $this->frags         = array();
          $this->fragCount     = 0;
          $this->lastFrag      = 0;
          $this->discontinuity = "";
          $this->InitDecoder();
        }

      function InitDecoder()
        {
          $this->audio             = false;
          $this->filesize          = 0;
          $this->video             = false;
          $this->prevTagSize       = 4;
          $this->tagHeaderLen      = 11;
          $this->baseTS            = INVALID_TIMESTAMP;
          $this->negTS             = INVALID_TIMESTAMP;
          $this->prevAudioTS       = INVALID_TIMESTAMP;
          $this->prevVideoTS       = INVALID_TIMESTAMP;
          $this->pAudioTagLen      = 0;
          $this->pVideoTagLen      = 0;
          $this->pAudioTagPos      = 0;
          $this->pVideoTagPos      = 0;
          $this->prevAVC_Header    = false;
          $this->prevAAC_Header    = false;
          $this->AVC_HeaderWritten = false;
          $this->AAC_HeaderWritten = false;
        }

      function GetManifest($cc, $manifest)
        {
          $status = $cc->get($manifest);
          if ($status == 403)
              LogError("Access Denied! Unable to download the manifest.");
          else if ($status != 200)
              LogError("Unable to download the manifest");
          $xml = simplexml_load_string(trim($cc->response));
          if (!$xml)
              LogError("Failed to load xml");
          $namespace = $xml->getDocNamespaces();
          $namespace = $namespace[''];
          $xml->registerXPathNamespace("ns", $namespace);
          return $xml;
        }

      function ParseManifest($cc, $parentManifest)
        {
          LogInfo("Processing manifest info....");
          $xml = $this->GetManifest($cc, $parentManifest);

          // Extract baseUrl from manifest url
          $baseUrl = $xml->xpath("/ns:manifest/ns:baseURL");
          if (isset($baseUrl[0]))
              $baseUrl = GetString($baseUrl[0]);
          else
            {
              $baseUrl = $parentManifest;
              if (strpos($baseUrl, '?') !== false)
                  $baseUrl = substr($baseUrl, 0, strpos($baseUrl, '?'));
              $baseUrl = substr($baseUrl, 0, strrpos($baseUrl, '/'));
            }

          $url = $xml->xpath("/ns:manifest/ns:media[@*]");
          if (isset($url[0]['href']))
            {
              $count = 1;
              foreach ($url as $childManifest)
                {
                  if (isset($childManifest['bitrate']))
                      $bitrate = floor(GetString($childManifest['bitrate']));
                  else
                      $bitrate = $count++;
                  $entry =& $childManifests[$bitrate];
                  $entry['bitrate'] = $bitrate;
                  $entry['url']     = AbsoluteUrl($baseUrl, GetString($childManifest['href']));
                  $entry['xml']     = $this->GetManifest($cc, $entry['url']);
                }
              unset($entry, $childManifest);
            }
          else
            {
              $childManifests[0]['bitrate'] = 0;
              $childManifests[0]['url']     = $parentManifest;
              $childManifests[0]['xml']     = $xml;
            }

          $count = 1;
          foreach ($childManifests as $childManifest)
            {
              $xml = $childManifest['xml'];

              // Extract baseUrl from manifest url
              $baseUrl = $xml->xpath("/ns:manifest/ns:baseURL");
              if (isset($baseUrl[0]))
                  $baseUrl = GetString($baseUrl[0]);
              else
                {
                  $baseUrl = $childManifest['url'];
                  if (strpos($baseUrl, '?') !== false)
                      $baseUrl = substr($baseUrl, 0, strpos($baseUrl, '?'));
                  $baseUrl = substr($baseUrl, 0, strrpos($baseUrl, '/'));
                }

              $streams = $xml->xpath("/ns:manifest/ns:media");
              foreach ($streams as $stream)
                {
                  $array = array();
                  foreach ($stream->attributes() as $k => $v)
                      $array[strtolower($k)] = GetString($v);
                  $array['metadata'] = GetString($stream->{'metadata'});
                  $stream            = $array;

                  if (isset($stream['bitrate']))
                      $bitrate = floor($stream['bitrate']);
                  else if ($childManifest['bitrate'] > 0)
                      $bitrate = $childManifest['bitrate'];
                  else
                      $bitrate = $count++;
                  while (isset($this->media[$bitrate]))
                      $bitrate++;
                  $streamId = isset($stream[strtolower('streamId')]) ? $stream[strtolower('streamId')] : "";
                  $mediaEntry =& $this->media[$bitrate];

                  $mediaEntry['baseUrl'] = $baseUrl;
                  $mediaEntry['url']     = $stream['url'];
                  if (isRtmpUrl($mediaEntry['baseUrl']) or isRtmpUrl($mediaEntry['url']))
                      LogError("Provided manifest is not a valid HDS manifest");

                  // Use embedded auth information when available
                  $idx = strpos($mediaEntry['url'], '?');
                  if ($idx !== false)
                    {
                      $mediaEntry['queryString'] = substr($mediaEntry['url'], $idx);
                      $mediaEntry['url']         = substr($mediaEntry['url'], 0, $idx);
                      if (strlen($this->auth) != 0 and strcmp($this->auth, $mediaEntry['queryString']) != 0)
                          LogDebug("Manifest overrides 'auth': " . $mediaEntry['queryString']);
                    }
                  else
                      $mediaEntry['queryString'] = $this->auth;

                  if (isset($stream[strtolower('bootstrapInfoId')]))
                      $bootstrap = $xml->xpath("/ns:manifest/ns:bootstrapInfo[@id='" . $stream[strtolower('bootstrapInfoId')] . "']");
                  else
                      $bootstrap = $xml->xpath("/ns:manifest/ns:bootstrapInfo");
                  if (isset($bootstrap[0]['url']))
                    {
                      $mediaEntry['bootstrapUrl'] = AbsoluteUrl($mediaEntry['baseUrl'], GetString($bootstrap[0]['url']));
                      if (strpos($mediaEntry['bootstrapUrl'], '?') === false)
                          $mediaEntry['bootstrapUrl'] .= $this->auth;
                    }
                  else
                      $mediaEntry['bootstrap'] = base64_decode(GetString($bootstrap[0]));
                  if (isset($stream['metadata']))
                      $mediaEntry['metadata'] = base64_decode($stream['metadata']);
                  else
                      $mediaEntry['metadata'] = "";
                }
              unset($mediaEntry, $childManifest);
            }

          // Available qualities
          $bitrates = array();
          if (!count($this->media))
              LogError("No media entry found");
          krsort($this->media, SORT_NUMERIC);
          LogDebug("Manifest Entries:\n");
          LogDebug(sprintf(" %-8s%s", "Bitrate", "URL"));
          for ($i = 0; $i media); $i++)
            {
              $key        = KeyName($this->media, $i);
              $bitrates[] = $key;
              LogDebug(sprintf(" %-8d%s", $key, $this->media[$key]['url']));
            }
          LogDebug("");
          LogInfo("Quality Selection:\n Available: " . implode(' ', $bitrates));

          // Quality selection
          if (is_numeric($this->quality) and isset($this->media[$this->quality]))
            {
              $key         = $this->quality;
              $this->media = $this->media[$key];
            }
          else
            {
              $this->quality = strtolower($this->quality);
              switch ($this->quality)
              {
                  case "low":
                      $this->quality = 2;
                      break;
                  case "medium":
                      $this->quality = 1;
                      break;
                  default:
                      $this->quality = 0;
              }
              while ($this->quality >= 0)
                {
                  $key = KeyName($this->media, $this->quality);
                  if ($key !== NULL)
                    {
                      $this->media = $this->media[$key];
                      break;
                    }
                  else
                      $this->quality -= 1;
                }
            }
          LogInfo(" Selected : " . $key);

          // Parse initial bootstrap info
          $this->baseUrl = $this->media['baseUrl'];
          if (isset($this->media['bootstrapUrl']))
            {
              $this->bootstrapUrl = $this->media['bootstrapUrl'];
              $this->UpdateBootstrapInfo($cc, $this->bootstrapUrl);
            }
          else
            {
              $bootstrapInfo = $this->media['bootstrap'];
              ReadBoxHeader($bootstrapInfo, $pos, $boxType, $boxSize);
              if ($boxType == "abst")
                  $this->ParseBootstrapBox($bootstrapInfo, $pos);
              else
                  LogError("Failed to parse bootstrap info");
            }
        }

      function UpdateBootstrapInfo($cc, $bootstrapUrl)
        {
          $fragNum = $this->fragCount;
          $retries = 0;

          // Backup original headers and add no-cache directive for fresh bootstrap info
          $headers       = $cc->headers;
          $cc->headers[] = "Cache-Control: no-cache";
          $cc->headers[] = "Pragma: no-cache";

          while (($fragNum == $this->fragCount) and ($retries fragCount);
              $status = $cc->get($bootstrapUrl);
              if ($status != 200)
                  LogError("Failed to refresh bootstrap info, Status: " . $status);
              $bootstrapInfo = $cc->response;
              ReadBoxHeader($bootstrapInfo, $bootstrapPos, $boxType, $boxSize);
              if ($boxType == "abst")
                  $this->ParseBootstrapBox($bootstrapInfo, $bootstrapPos);
              else
                  LogError("Failed to parse bootstrap info");
              LogDebug("Update complete, Available fragments: " . $this->fragCount);
              if ($fragNum == $this->fragCount)
                {
                  LogInfo("Updating bootstrap info, Retries: " . ++$retries, true);
                  usleep(4000000);
                }
            }

          // Restore original headers
          $cc->headers = $headers;
        }

      function ParseBootstrapBox($bootstrapInfo, $pos)
        {
          $version          = ReadByte($bootstrapInfo, $pos);
          $flags            = ReadInt24($bootstrapInfo, $pos + 1);
          $bootstrapVersion = ReadInt32($bootstrapInfo, $pos + 4);
          $byte             = ReadByte($bootstrapInfo, $pos + 8);
          $profile          = ($byte & 0xC0) >> 6;
          if (($byte & 0x20) >> 5)
            {
              $this->live     = true;
              $this->metadata = false;
            }
          $update = ($byte & 0x10) >> 4;
          if (!$update)
            {
              $this->segTable  = array();
              $this->fragTable = array();
            }
          $timescale           = ReadInt32($bootstrapInfo, $pos + 9);
          $currentMediaTime    = ReadInt64($bootstrapInfo, $pos + 13);
          $smpteTimeCodeOffset = ReadInt64($bootstrapInfo, $pos + 21);
          $pos += 29;
          $movieIdentifier  = ReadString($bootstrapInfo, $pos);
          $serverEntryCount = ReadByte($bootstrapInfo, $pos++);
          for ($i = 0; $i < $serverEntryCount; $i++)
              $serverEntryTable[$i] = ReadString($bootstrapInfo, $pos);
          $qualityEntryCount = ReadByte($bootstrapInfo, $pos++);
          for ($i = 0; $i < $qualityEntryCount; $i++)
              $qualityEntryTable[$i] = ReadString($bootstrapInfo, $pos);
          $drmData          = ReadString($bootstrapInfo, $pos);
          $metadata         = ReadString($bootstrapInfo, $pos);
          $segRunTableCount = ReadByte($bootstrapInfo, $pos++);
          LogDebug(sprintf("%s:", "Segment Tables"));
          for ($i = 0; $i ParseAsrtBox($bootstrapInfo, $pos);
              $pos += $boxSize;
            }
          $fragRunTableCount = ReadByte($bootstrapInfo, $pos++);
          LogDebug(sprintf("%s:", "Fragment Tables"));
          for ($i = 0; $i ParseAfrtBox($bootstrapInfo, $pos);
              $pos += $boxSize;
            }
          $this->segTable  = array_replace($this->segTable, $segTable[0]);
          $this->fragTable = array_replace($this->fragTable, $fragTable[0]);
          $this->ParseSegAndFragTable();
        }

      function ParseAsrtBox($asrt, $pos)
        {
          $segTable          = array();
          $version           = ReadByte($asrt, $pos);
          $flags             = ReadInt24($asrt, $pos + 1);
          $qualityEntryCount = ReadByte($asrt, $pos + 4);
          $pos += 5;
          for ($i = 0; $i < $qualityEntryCount; $i++)
              $qualitySegmentUrlModifiers[$i] = ReadString($asrt, $pos);
          $segCount = ReadInt32($asrt, $pos);
          $pos += 4;
          LogDebug(sprintf(" %-8s%-10s", "Number", "Fragments"));
          for ($i = 0; $i < $segCount; $i++)
            {
              $firstSegment = ReadInt32($asrt, $pos);
              $segEntry =& $segTable[$firstSegment];
              $segEntry['firstSegment']        = $firstSegment;
              $segEntry['fragmentsPerSegment'] = ReadInt32($asrt, $pos + 4);
              if ($segEntry['fragmentsPerSegment'] & 0x80000000)
                  $segEntry['fragmentsPerSegment'] = 0;
              $pos += 8;
            }
          unset($segEntry);
          foreach ($segTable as $segEntry)
              LogDebug(sprintf(" %-8s%-10s", $segEntry['firstSegment'], $segEntry['fragmentsPerSegment']));
          LogDebug("");
          return $segTable;
        }

      function ParseAfrtBox($afrt, $pos)
        {
          $fragTable         = array();
          $version           = ReadByte($afrt, $pos);
          $flags             = ReadInt24($afrt, $pos + 1);
          $timescale         = ReadInt32($afrt, $pos + 4);
          $qualityEntryCount = ReadByte($afrt, $pos + 8);
          $pos += 9;
          for ($i = 0; $i < $qualityEntryCount; $i++)
              $qualitySegmentUrlModifiers[$i] = ReadString($afrt, $pos);
          $fragEntries = ReadInt32($afrt, $pos);
          $pos += 4;
          LogDebug(sprintf(" %-12s%-16s%-16s%-16s", "Number", "Timestamp", "Duration", "Discontinuity"));
          for ($i = 0; $i segTable);
          $lastSegment   = end($this->segTable);
          $firstFragment = reset($this->fragTable);
          $lastFragment  = end($this->fragTable);

          // Check if live stream is still live
          if (($lastFragment['fragmentDuration'] == 0) and ($lastFragment['discontinuityIndicator'] == 0))
            {
              $this->live = false;
              array_pop($this->fragTable);
              $lastFragment = end($this->fragTable);
            }

          // Count total fragments by adding all entries in compactly coded segment table
          $invalidFragCount = false;
          $prev             = reset($this->segTable);
          $this->fragCount  = $prev['fragmentsPerSegment'];
          while ($current = next($this->segTable))
            {
              $this->fragCount += ($current['firstSegment'] - $prev['firstSegment'] - 1) * $prev['fragmentsPerSegment'];
              $this->fragCount += $current['fragmentsPerSegment'];
              $prev = $current;
            }
          if (!($this->fragCount & 0x80000000))
              $this->fragCount += $firstFragment['firstFragment'] - 1;
          if ($this->fragCount & 0x80000000)
            {
              $this->fragCount  = 0;
              $invalidFragCount = true;
            }
          if ($this->fragCount fragCount = $lastFragment['firstFragment

 

image

 

4.) You should now have a single .FLV video file with all the fragments joined together. If you use a media player that can play FLV video, such as VLC you can stop here. However, if you want to make the video a slightly more standard .MP4 format, you can convert it using FFMPEG, like this:

ffmpeg -i The_Music_Mash_Mustard_TV_video_1200_1080Seg1_Frag.flv -vcodec copy -acodec copy output.mp4
 
Posted in Uncategorized | Tagged , | Leave a comment

Underwater Effects

There’s lots of resources that describe techniques for realistic rendering of water surfaces. Unity comes supplied with several standard water assets, and there are some excellent descriptions of more advanced water effects at scrawkblog to generate spectacular ocean surfaces such as this:

Or Martin Upitis’ blender file: http://www.blendswap.com/blends/view/68857

However, I see much less information and examples of creating underwater effects, so here’s some notes on how I’ve recently approached different elements of making an underwater game scene.

 

a.) Fog

However clear the water in which you’re swimming, visibility is always going to be less than that in air, and global fog is a pretty easy way to achieve this. I chose a grey-blue colour (say, RGB 60,100,120) with Exp2 fog mode, and density of around 0.005 – adjustable depending on how murky you want it.

 

b.) Sea Floor

I got this nice diffuse tiling sand texture from CGTextures.

image

I like this image because instead of having a detailed grain texture, it has those nice smooth marks that you expect from sand that (at least at some point in time) has had waves wash over it. I then generated a normal map from the texture using the GIMP NormalMap plugin (not to be confused with the BumpMap plugin):

image

Then I created a simple Quad and applied a bumped diffuse material to it using these two textures. For more detail you could of course use a sculpted mesh or terrain, but a flat plane was fine for me to start with. Here it is, along with the underwater fog defined previously:

image

 

c.) Caustics

Caustics are the patterns of light you see projected onto surfaces under the water.

http://habibs.files.wordpress.com/2008/04/caustics_rendering_example.jpg

Caustics are easily simulated by using a projector to show an animated texture. The projector should be set to point straight down (i.e. rotation of 90 degrees about the X axis), with an orthographic projection. Then, attach the following script to the projector, which will allow it to cycle through a series of frames of animation:

using UnityEngine;
using System.Collections;

public class AnimatedProjector : MonoBehaviour
{
    public float fps = 30.0f;
    public Texture2D[] frames;
    private int frameIndex;
    private Projector projector;

    void Start()
    {
        projector = GetComponent();
        InvokeRepeating("NextFrame", 0, 1 / fps);
    }

    void NextFrame()
    {
        projector.material.SetTexture("_MainTex", frames[frameIndex]);
        frameIndex = (frameIndex + 1) % frames.Length;
    }
}

To create the frames of caustic animation, you can use a tool such as http://www.dualheights.se/caustics/ (free for non-commercial use). I find that you’ll need to generate somewhere between 30 – 60 frames to get a convincing animation. The frames should look something like this:

image

Assign these textures to the frames[] array of the AnimatedProjector script above, and use a shader with an additive blend mode so that the white parts of the texture lighten the surface onto which they are projected. The caustic generator above creates tileable images, so you can position a grid of projectors next to each other if you want to cover a large area (that’s why you should also use an orthographic projection, to ensure the images don’t get distorted at the edges, which would mess up the tiling).

 

d.) Bubbles

Although I’ve personally never seen anything on the seabed that generates a constant stream of bubbles, it seems to be standard fare in computer games for conveying a sense of being underwater.

https://i2.wp.com/critical-gaming.com/storage/coin6.4.png

These are easy to achieve using a particle emitter to create billboard 2D sprites of a simple bubble texture, drawn using an additive shader. Here’s one example of a suitable bubble texture:

image

If you want to get fancy, you can create a slightly animated texture or apply random forces to make the bubbles “wobble” as they rise to the surface, but generally it’s acceptable to emit them at an angle close to vertical (here I use a cone emitter with radius of 5 degrees). You can add some slight randomness to the size and speed of bubbles too. I don’t understand the exact, complicated physics, but I believe bubbles grow larger as they rise towards the surface, and also large bubbles rise faster than small bubbles, which I’ve implemented through the “Size over lifetime” and “Size by speed” curves. Finally, I make my bubbles “fade out” using an alpha curve in “Color over lifetime”; I’m not sure this is realistic, but it prevents bubbles suddenly “popping” while still underwater. You could instead adjust the particle lifetime to ensure that bubbles just reach the surface before being destroyed. Here’s the complete settings I used:

image

and here’s the bubble stream they create:

image

 

e.) Flora, Fauna, and Furniture

The appropriate props to add to an underwater scene will depend on whether it’s intended to be tropical or polar, shallow or deep water, however, once again sticking with what people expect to find in an underwater game, I added a few rocks, corals, fish, and a sunken rowing boat. Goodness knows whether any of these are authentic or not, but at least they make the scene more interesting. You can find lots of suitable assets, many of them free, on the Unity Asset Store.

 

The Finished Scene

Here’s the final scene – nothing particularly innovative or hardware-taxing, and certainly suitable for mobile devices, but hopefully conveying what players expect from an underwater scene:

image

 

And a video to show the effect of the caustics in action…

Posted in Game Dev | Tagged , , , | Leave a comment