Grabbing and Converting Adobe Streaming Video Fragments

I was recently interviewed by Mustard TV for a feature on the Norwich Sound & Vision festival, in which I’ve been involved. The feature was broadcast on Freeview cable TV but I can’t pick up the channel where I live, so I wanted to grab a copy off the website instead. They deliver videos using Adobe’s adaptive streaming protocol, which results in videos being sent in many fragments, of potentially different bitrates.

Here’s a little guide of what I had to do to save, join, and then convert the fragments into a single .MP4 video file.

 

1.) First, I went to the page in which the video was embedded and played the video the whole way through.

image

IMPORTANT NOTE:  Since Adobe’s streaming media server chunks videos into many different fragments, you need to watch the entire video the whole way through to make sure all the fragments are requested and downloaded.

 

2.) Once the video has ended, I used the media tab of the FlashGot FireFox extension to save all the video fragments to a local folder.

image

 

IMPORTANT NOTE: Due to the nature of adaptive bitrate streaming, you’ll see that the resolution of some of the fragments above varies depending on what my internet speed was like at the time. (Fragments 4 and 5 are only 600×640 resolution rather than 1200×1080 – I must’ve had some sort of spike at that point). To join a single video, you’ll want to ensure that you end up with a complete set of fragments at the same resolution. For fragments 4 and 5 above, I simply replayed that part of the video and it re-requested the file at a better resolution – you could also probably copy the URL template and specify the resolution of the fragment you want.

 

3.) To join the fragments together, I then used a PHP script which I found here. I’m reproducing it below for convenience.

To use the script, start a new command prompt and call it as follows, with the full filename omitting the fragment number:

php AdobeHDS.php --fragments The_Music_Mash_Mustard_TV_video_1200_1080Seg1_Frag

 

image

 

AdobeHDS.php

 1)
            {
              $paramSwitch = false;
              for ($i = 1; $i error("[param] expected after '$paramSwitch' switch (" . self::$ACCEPTED[1][$paramSwitch] . ')');
                  else if (!$paramSwitch and !$isSwitch)
                    {
                      if ($handleUnknown)
                          $this->params['unknown'][] = $arg;
                      else
                          $this->error("'$arg' is an invalid option, use --help to display valid switches.");
                    }
                  else if (!$paramSwitch and $isSwitch)
                    {
                      if (isset($this->params[$arg]))
                          $this->error("'$arg' switch can't occur more than once");

                      $this->params[$arg] = true;
                      if (isset(self::$ACCEPTED[1][$arg]))
                          $paramSwitch = $arg;
                      else if (!isset(self::$ACCEPTED[0][$arg]))
                          $this->error("there's no '$arg' switch, use --help to display all switches.");
                    }
                  else if ($paramSwitch and !$isSwitch)
                    {
                      $this->params[$paramSwitch] = $arg;
                      $paramSwitch                = false;
                    }
                }
            }

          // Final check
          foreach ($this->params as $k => $v)
              if (isset(self::$ACCEPTED[1][$k]) and $v === true)
                  $this->error("[param] expected after '$k' switch (" . self::$ACCEPTED[1][$k] . ')');
        }

      function displayHelp()
        {
          LogInfo("You can use script with following switches:\n");
          foreach (self::$ACCEPTED[0] as $key => $value)
              LogInfo(sprintf(" --%-17s %s", $key, $value));
          foreach (self::$ACCEPTED[1] as $key => $value)
              LogInfo(sprintf(" --%-9s%-8s %s", $key, " [param]", $value));
        }

      function error($msg)
        {
          LogError($msg);
        }

      function getParam($name)
        {
          if (isset($this->params[$name]))
              return $this->params[$name];
          else
              return false;
        }
    }

  class cURL
    {
      var $headers, $user_agent, $compression, $cookie_file;
      var $active, $cert_check, $fragProxy, $maxSpeed, $proxy, $response;
      var $mh, $ch, $mrc;
      static $ref = 0;

      function cURL($cookies = true, $cookie = 'Cookies.txt', $compression = 'gzip', $proxy = '')
        {
          $this->headers     = $this->headers();
          $this->user_agent  = 'Mozilla/5.0 (Windows NT 5.1; rv:26.0) Gecko/20100101 Firefox/26.0';
          $this->compression = $compression;
          $this->cookies     = $cookies;
          if ($this->cookies == true)
              $this->cookie($cookie);
          $this->cert_check = false;
          $this->fragProxy  = false;
          $this->maxSpeed   = 0;
          $this->proxy      = $proxy;
          self::$ref++;
        }

      function __destruct()
        {
          $this->stopDownloads();
          if ((self::$ref cookie_file))
              unlink($this->cookie_file);
          self::$ref--;
        }

      function headers()
        {
          $headers[] = 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8';
          $headers[] = 'Connection: Keep-Alive';
          return $headers;
        }

      function cookie($cookie_file)
        {
          if (file_exists($cookie_file))
              $this->cookie_file = $cookie_file;
          else
            {
              $file = fopen($cookie_file, 'w') or $this->error('The cookie file could not be opened. Make sure this directory has the correct permissions.');
              $this->cookie_file = $cookie_file;
              fclose($file);
            }
        }

      function get($url)
        {
          $process = curl_init($url);
          $options = array(
              CURLOPT_HTTPHEADER => $this->headers,
              CURLOPT_HEADER => 0,
              CURLOPT_USERAGENT => $this->user_agent,
              CURLOPT_ENCODING => $this->compression,
              CURLOPT_TIMEOUT => 30,
              CURLOPT_RETURNTRANSFER => 1,
              CURLOPT_FOLLOWLOCATION => 1
          );
          curl_setopt_array($process, $options);
          if (!$this->cert_check)
              curl_setopt($process, CURLOPT_SSL_VERIFYPEER, false);
          if ($this->cookies == true)
            {
              curl_setopt($process, CURLOPT_COOKIEFILE, $this->cookie_file);
              curl_setopt($process, CURLOPT_COOKIEJAR, $this->cookie_file);
            }
          if ($this->proxy)
              $this->setProxy($process, $this->proxy);
          $this->response = curl_exec($process);
          if ($this->response !== false)
              $status = curl_getinfo($process, CURLINFO_HTTP_CODE);
          curl_close($process);
          if (isset($status))
              return $status;
          else
              return false;
        }

      function post($url, $data)
        {
          $process   = curl_init($url);
          $headers   = $this->headers;
          $headers[] = 'Content-Type: application/x-www-form-urlencoded;charset=UTF-8';
          $options   = array(
              CURLOPT_HTTPHEADER => $headers,
              CURLOPT_HEADER => 1,
              CURLOPT_USERAGENT => $this->user_agent,
              CURLOPT_ENCODING => $this->compression,
              CURLOPT_TIMEOUT => 30,
              CURLOPT_RETURNTRANSFER => 1,
              CURLOPT_FOLLOWLOCATION => 1,
              CURLOPT_POST => 1,
              CURLOPT_POSTFIELDS => $data
          );
          curl_setopt_array($process, $options);
          if (!$this->cert_check)
              curl_setopt($process, CURLOPT_SSL_VERIFYPEER, false);
          if ($this->cookies == true)
            {
              curl_setopt($process, CURLOPT_COOKIEFILE, $this->cookie_file);
              curl_setopt($process, CURLOPT_COOKIEJAR, $this->cookie_file);
            }
          if ($this->proxy)
              $this->setProxy($process, $this->proxy);
          $return = curl_exec($process);
          curl_close($process);
          return $return;
        }

      function setProxy(&$process, $proxy)
        {
          $type      = "";
          $separator = strpos($proxy, "://");
          if ($separator !== false)
            {
              $type  = strtolower(substr($proxy, 0, $separator));
              $proxy = substr($proxy, $separator + 3);
            }
          switch ($type)
          {
              case "socks4":
                  $type = CURLPROXY_SOCKS4;
                  break;
              case "socks5":
                  $type = CURLPROXY_SOCKS5;
                  break;
              default:
                  $type = CURLPROXY_HTTP;
          }
          curl_setopt($process, CURLOPT_PROXY, $proxy);
          curl_setopt($process, CURLOPT_PROXYTYPE, $type);
        }

      function addDownload($url, $id)
        {
          if (!isset($this->mh))
              $this->mh = curl_multi_init();
          if (isset($this->ch[$id]))
              return false;
          $download =& $this->ch[$id];
          $download['id']  = $id;
          $download['url'] = $url;
          $download['ch']  = curl_init($url);
          $options         = array(
              CURLOPT_HTTPHEADER => $this->headers,
              CURLOPT_HEADER => 0,
              CURLOPT_USERAGENT => $this->user_agent,
              CURLOPT_ENCODING => $this->compression,
              CURLOPT_LOW_SPEED_LIMIT => 1024,
              CURLOPT_LOW_SPEED_TIME => 10,
              CURLOPT_BINARYTRANSFER => 1,
              CURLOPT_RETURNTRANSFER => 1,
              CURLOPT_FOLLOWLOCATION => 1
          );
          curl_setopt_array($download['ch'], $options);
          if (!$this->cert_check)
              curl_setopt($download['ch'], CURLOPT_SSL_VERIFYPEER, false);
          if ($this->cookies == true)
            {
              curl_setopt($download['ch'], CURLOPT_COOKIEFILE, $this->cookie_file);
              curl_setopt($download['ch'], CURLOPT_COOKIEJAR, $this->cookie_file);
            }
          if ($this->fragProxy and $this->proxy)
              $this->setProxy($download['ch'], $this->proxy);
          if ($this->maxSpeed > 0)
              curl_setopt($process, CURLOPT_MAX_RECV_SPEED_LARGE, $this->maxSpeed);
          curl_multi_add_handle($this->mh, $download['ch']);
          do
            {
              $this->mrc = curl_multi_exec($this->mh, $this->active);
            } while ($this->mrc == CURLM_CALL_MULTI_PERFORM);
          return true;
        }

      function checkDownloads()
        {
          if (isset($this->mh))
            {
              curl_multi_select($this->mh);
              $this->mrc = curl_multi_exec($this->mh, $this->active);
              if ($this->mrc != CURLM_OK)
                  return false;
              while ($info = curl_multi_info_read($this->mh))
                {
                  foreach ($this->ch as $download)
                      if ($download['ch'] == $info['handle'])
                          break;
                  $array['id']  = $download['id'];
                  $array['url'] = $download['url'];
                  $info         = curl_getinfo($download['ch']);
                  if ($info['http_code'] == 0)
                    {
                      /* if curl fails due to network connectivity issues or some other reason it's *
                       * better to add some delay before next try to avoid busy loop.               */
                      LogDebug("Fragment " . $download['id'] . ": " . curl_error($download['ch']));
                      usleep(1000000);
                      $array['status']   = false;
                      $array['response'] = "";
                    }
                  else if ($info['http_code'] == 200)
                    {
                      if ($info['size_download'] >= $info['download_content_length'])
                        {
                          $array['status']   = $info['http_code'];
                          $array['response'] = curl_multi_getcontent($download['ch']);
                        }
                      else
                        {
                          $array['status']   = false;
                          $array['response'] = "";
                        }
                    }
                  else
                    {
                      $array['status']   = $info['http_code'];
                      $array['response'] = curl_multi_getcontent($download['ch']);
                    }
                  $downloads[] = $array;
                  curl_multi_remove_handle($this->mh, $download['ch']);
                  curl_close($download['ch']);
                  unset($this->ch[$download['id']]);
                }
              if (isset($downloads) and (count($downloads) > 0))
                  return $downloads;
            }
          return false;
        }

      function stopDownloads()
        {
          if (isset($this->mh))
            {
              if (isset($this->ch))
                {
                  foreach ($this->ch as $download)
                    {
                      curl_multi_remove_handle($this->mh, $download['ch']);
                      curl_close($download['ch']);
                    }
                  unset($this->ch);
                }
              curl_multi_close($this->mh);
              unset($this->mh);
            }
        }

      function error($error)
        {
          LogError("cURL Error : $error");
        }
    }

  class F4F
    {
      var $audio, $auth, $baseFilename, $baseTS, $bootstrapUrl, $baseUrl, $debug, $duration, $fileCount, $filesize, $fixWindow;
      var $format, $live, $media, $metadata, $outDir, $outFile, $parallel, $play, $processed, $quality, $rename, $video;
      var $prevTagSize, $tagHeaderLen;
      var $segTable, $fragTable, $segNum, $fragNum, $frags, $fragCount, $lastFrag, $fragUrl, $discontinuity;
      var $prevAudioTS, $prevVideoTS, $pAudioTagLen, $pVideoTagLen, $pAudioTagPos, $pVideoTagPos;
      var $prevAVC_Header, $prevAAC_Header, $AVC_HeaderWritten, $AAC_HeaderWritten;

      function __construct()
        {
          $this->auth          = "";
          $this->baseFilename  = "";
          $this->bootstrapUrl  = "";
          $this->debug         = false;
          $this->duration      = 0;
          $this->fileCount     = 1;
          $this->fixWindow     = 1000;
          $this->format        = "";
          $this->live          = false;
          $this->metadata      = true;
          $this->outDir        = "";
          $this->outFile       = "";
          $this->parallel      = 8;
          $this->play          = false;
          $this->processed     = false;
          $this->quality       = "high";
          $this->rename        = false;
          $this->segTable      = array();
          $this->fragTable     = array();
          $this->segStart      = false;
          $this->fragStart     = false;
          $this->frags         = array();
          $this->fragCount     = 0;
          $this->lastFrag      = 0;
          $this->discontinuity = "";
          $this->InitDecoder();
        }

      function InitDecoder()
        {
          $this->audio             = false;
          $this->filesize          = 0;
          $this->video             = false;
          $this->prevTagSize       = 4;
          $this->tagHeaderLen      = 11;
          $this->baseTS            = INVALID_TIMESTAMP;
          $this->negTS             = INVALID_TIMESTAMP;
          $this->prevAudioTS       = INVALID_TIMESTAMP;
          $this->prevVideoTS       = INVALID_TIMESTAMP;
          $this->pAudioTagLen      = 0;
          $this->pVideoTagLen      = 0;
          $this->pAudioTagPos      = 0;
          $this->pVideoTagPos      = 0;
          $this->prevAVC_Header    = false;
          $this->prevAAC_Header    = false;
          $this->AVC_HeaderWritten = false;
          $this->AAC_HeaderWritten = false;
        }

      function GetManifest($cc, $manifest)
        {
          $status = $cc->get($manifest);
          if ($status == 403)
              LogError("Access Denied! Unable to download the manifest.");
          else if ($status != 200)
              LogError("Unable to download the manifest");
          $xml = simplexml_load_string(trim($cc->response));
          if (!$xml)
              LogError("Failed to load xml");
          $namespace = $xml->getDocNamespaces();
          $namespace = $namespace[''];
          $xml->registerXPathNamespace("ns", $namespace);
          return $xml;
        }

      function ParseManifest($cc, $parentManifest)
        {
          LogInfo("Processing manifest info....");
          $xml = $this->GetManifest($cc, $parentManifest);

          // Extract baseUrl from manifest url
          $baseUrl = $xml->xpath("/ns:manifest/ns:baseURL");
          if (isset($baseUrl[0]))
              $baseUrl = GetString($baseUrl[0]);
          else
            {
              $baseUrl = $parentManifest;
              if (strpos($baseUrl, '?') !== false)
                  $baseUrl = substr($baseUrl, 0, strpos($baseUrl, '?'));
              $baseUrl = substr($baseUrl, 0, strrpos($baseUrl, '/'));
            }

          $url = $xml->xpath("/ns:manifest/ns:media[@*]");
          if (isset($url[0]['href']))
            {
              $count = 1;
              foreach ($url as $childManifest)
                {
                  if (isset($childManifest['bitrate']))
                      $bitrate = floor(GetString($childManifest['bitrate']));
                  else
                      $bitrate = $count++;
                  $entry =& $childManifests[$bitrate];
                  $entry['bitrate'] = $bitrate;
                  $entry['url']     = AbsoluteUrl($baseUrl, GetString($childManifest['href']));
                  $entry['xml']     = $this->GetManifest($cc, $entry['url']);
                }
              unset($entry, $childManifest);
            }
          else
            {
              $childManifests[0]['bitrate'] = 0;
              $childManifests[0]['url']     = $parentManifest;
              $childManifests[0]['xml']     = $xml;
            }

          $count = 1;
          foreach ($childManifests as $childManifest)
            {
              $xml = $childManifest['xml'];

              // Extract baseUrl from manifest url
              $baseUrl = $xml->xpath("/ns:manifest/ns:baseURL");
              if (isset($baseUrl[0]))
                  $baseUrl = GetString($baseUrl[0]);
              else
                {
                  $baseUrl = $childManifest['url'];
                  if (strpos($baseUrl, '?') !== false)
                      $baseUrl = substr($baseUrl, 0, strpos($baseUrl, '?'));
                  $baseUrl = substr($baseUrl, 0, strrpos($baseUrl, '/'));
                }

              $streams = $xml->xpath("/ns:manifest/ns:media");
              foreach ($streams as $stream)
                {
                  $array = array();
                  foreach ($stream->attributes() as $k => $v)
                      $array[strtolower($k)] = GetString($v);
                  $array['metadata'] = GetString($stream->{'metadata'});
                  $stream            = $array;

                  if (isset($stream['bitrate']))
                      $bitrate = floor($stream['bitrate']);
                  else if ($childManifest['bitrate'] > 0)
                      $bitrate = $childManifest['bitrate'];
                  else
                      $bitrate = $count++;
                  while (isset($this->media[$bitrate]))
                      $bitrate++;
                  $streamId = isset($stream[strtolower('streamId')]) ? $stream[strtolower('streamId')] : "";
                  $mediaEntry =& $this->media[$bitrate];

                  $mediaEntry['baseUrl'] = $baseUrl;
                  $mediaEntry['url']     = $stream['url'];
                  if (isRtmpUrl($mediaEntry['baseUrl']) or isRtmpUrl($mediaEntry['url']))
                      LogError("Provided manifest is not a valid HDS manifest");

                  // Use embedded auth information when available
                  $idx = strpos($mediaEntry['url'], '?');
                  if ($idx !== false)
                    {
                      $mediaEntry['queryString'] = substr($mediaEntry['url'], $idx);
                      $mediaEntry['url']         = substr($mediaEntry['url'], 0, $idx);
                      if (strlen($this->auth) != 0 and strcmp($this->auth, $mediaEntry['queryString']) != 0)
                          LogDebug("Manifest overrides 'auth': " . $mediaEntry['queryString']);
                    }
                  else
                      $mediaEntry['queryString'] = $this->auth;

                  if (isset($stream[strtolower('bootstrapInfoId')]))
                      $bootstrap = $xml->xpath("/ns:manifest/ns:bootstrapInfo[@id='" . $stream[strtolower('bootstrapInfoId')] . "']");
                  else
                      $bootstrap = $xml->xpath("/ns:manifest/ns:bootstrapInfo");
                  if (isset($bootstrap[0]['url']))
                    {
                      $mediaEntry['bootstrapUrl'] = AbsoluteUrl($mediaEntry['baseUrl'], GetString($bootstrap[0]['url']));
                      if (strpos($mediaEntry['bootstrapUrl'], '?') === false)
                          $mediaEntry['bootstrapUrl'] .= $this->auth;
                    }
                  else
                      $mediaEntry['bootstrap'] = base64_decode(GetString($bootstrap[0]));
                  if (isset($stream['metadata']))
                      $mediaEntry['metadata'] = base64_decode($stream['metadata']);
                  else
                      $mediaEntry['metadata'] = "";
                }
              unset($mediaEntry, $childManifest);
            }

          // Available qualities
          $bitrates = array();
          if (!count($this->media))
              LogError("No media entry found");
          krsort($this->media, SORT_NUMERIC);
          LogDebug("Manifest Entries:\n");
          LogDebug(sprintf(" %-8s%s", "Bitrate", "URL"));
          for ($i = 0; $i media); $i++)
            {
              $key        = KeyName($this->media, $i);
              $bitrates[] = $key;
              LogDebug(sprintf(" %-8d%s", $key, $this->media[$key]['url']));
            }
          LogDebug("");
          LogInfo("Quality Selection:\n Available: " . implode(' ', $bitrates));

          // Quality selection
          if (is_numeric($this->quality) and isset($this->media[$this->quality]))
            {
              $key         = $this->quality;
              $this->media = $this->media[$key];
            }
          else
            {
              $this->quality = strtolower($this->quality);
              switch ($this->quality)
              {
                  case "low":
                      $this->quality = 2;
                      break;
                  case "medium":
                      $this->quality = 1;
                      break;
                  default:
                      $this->quality = 0;
              }
              while ($this->quality >= 0)
                {
                  $key = KeyName($this->media, $this->quality);
                  if ($key !== NULL)
                    {
                      $this->media = $this->media[$key];
                      break;
                    }
                  else
                      $this->quality -= 1;
                }
            }
          LogInfo(" Selected : " . $key);

          // Parse initial bootstrap info
          $this->baseUrl = $this->media['baseUrl'];
          if (isset($this->media['bootstrapUrl']))
            {
              $this->bootstrapUrl = $this->media['bootstrapUrl'];
              $this->UpdateBootstrapInfo($cc, $this->bootstrapUrl);
            }
          else
            {
              $bootstrapInfo = $this->media['bootstrap'];
              ReadBoxHeader($bootstrapInfo, $pos, $boxType, $boxSize);
              if ($boxType == "abst")
                  $this->ParseBootstrapBox($bootstrapInfo, $pos);
              else
                  LogError("Failed to parse bootstrap info");
            }
        }

      function UpdateBootstrapInfo($cc, $bootstrapUrl)
        {
          $fragNum = $this->fragCount;
          $retries = 0;

          // Backup original headers and add no-cache directive for fresh bootstrap info
          $headers       = $cc->headers;
          $cc->headers[] = "Cache-Control: no-cache";
          $cc->headers[] = "Pragma: no-cache";

          while (($fragNum == $this->fragCount) and ($retries fragCount);
              $status = $cc->get($bootstrapUrl);
              if ($status != 200)
                  LogError("Failed to refresh bootstrap info, Status: " . $status);
              $bootstrapInfo = $cc->response;
              ReadBoxHeader($bootstrapInfo, $bootstrapPos, $boxType, $boxSize);
              if ($boxType == "abst")
                  $this->ParseBootstrapBox($bootstrapInfo, $bootstrapPos);
              else
                  LogError("Failed to parse bootstrap info");
              LogDebug("Update complete, Available fragments: " . $this->fragCount);
              if ($fragNum == $this->fragCount)
                {
                  LogInfo("Updating bootstrap info, Retries: " . ++$retries, true);
                  usleep(4000000);
                }
            }

          // Restore original headers
          $cc->headers = $headers;
        }

      function ParseBootstrapBox($bootstrapInfo, $pos)
        {
          $version          = ReadByte($bootstrapInfo, $pos);
          $flags            = ReadInt24($bootstrapInfo, $pos + 1);
          $bootstrapVersion = ReadInt32($bootstrapInfo, $pos + 4);
          $byte             = ReadByte($bootstrapInfo, $pos + 8);
          $profile          = ($byte & 0xC0) >> 6;
          if (($byte & 0x20) >> 5)
            {
              $this->live     = true;
              $this->metadata = false;
            }
          $update = ($byte & 0x10) >> 4;
          if (!$update)
            {
              $this->segTable  = array();
              $this->fragTable = array();
            }
          $timescale           = ReadInt32($bootstrapInfo, $pos + 9);
          $currentMediaTime    = ReadInt64($bootstrapInfo, $pos + 13);
          $smpteTimeCodeOffset = ReadInt64($bootstrapInfo, $pos + 21);
          $pos += 29;
          $movieIdentifier  = ReadString($bootstrapInfo, $pos);
          $serverEntryCount = ReadByte($bootstrapInfo, $pos++);
          for ($i = 0; $i < $serverEntryCount; $i++)
              $serverEntryTable[$i] = ReadString($bootstrapInfo, $pos);
          $qualityEntryCount = ReadByte($bootstrapInfo, $pos++);
          for ($i = 0; $i < $qualityEntryCount; $i++)
              $qualityEntryTable[$i] = ReadString($bootstrapInfo, $pos);
          $drmData          = ReadString($bootstrapInfo, $pos);
          $metadata         = ReadString($bootstrapInfo, $pos);
          $segRunTableCount = ReadByte($bootstrapInfo, $pos++);
          LogDebug(sprintf("%s:", "Segment Tables"));
          for ($i = 0; $i ParseAsrtBox($bootstrapInfo, $pos);
              $pos += $boxSize;
            }
          $fragRunTableCount = ReadByte($bootstrapInfo, $pos++);
          LogDebug(sprintf("%s:", "Fragment Tables"));
          for ($i = 0; $i ParseAfrtBox($bootstrapInfo, $pos);
              $pos += $boxSize;
            }
          $this->segTable  = array_replace($this->segTable, $segTable[0]);
          $this->fragTable = array_replace($this->fragTable, $fragTable[0]);
          $this->ParseSegAndFragTable();
        }

      function ParseAsrtBox($asrt, $pos)
        {
          $segTable          = array();
          $version           = ReadByte($asrt, $pos);
          $flags             = ReadInt24($asrt, $pos + 1);
          $qualityEntryCount = ReadByte($asrt, $pos + 4);
          $pos += 5;
          for ($i = 0; $i < $qualityEntryCount; $i++)
              $qualitySegmentUrlModifiers[$i] = ReadString($asrt, $pos);
          $segCount = ReadInt32($asrt, $pos);
          $pos += 4;
          LogDebug(sprintf(" %-8s%-10s", "Number", "Fragments"));
          for ($i = 0; $i < $segCount; $i++)
            {
              $firstSegment = ReadInt32($asrt, $pos);
              $segEntry =& $segTable[$firstSegment];
              $segEntry['firstSegment']        = $firstSegment;
              $segEntry['fragmentsPerSegment'] = ReadInt32($asrt, $pos + 4);
              if ($segEntry['fragmentsPerSegment'] & 0x80000000)
                  $segEntry['fragmentsPerSegment'] = 0;
              $pos += 8;
            }
          unset($segEntry);
          foreach ($segTable as $segEntry)
              LogDebug(sprintf(" %-8s%-10s", $segEntry['firstSegment'], $segEntry['fragmentsPerSegment']));
          LogDebug("");
          return $segTable;
        }

      function ParseAfrtBox($afrt, $pos)
        {
          $fragTable         = array();
          $version           = ReadByte($afrt, $pos);
          $flags             = ReadInt24($afrt, $pos + 1);
          $timescale         = ReadInt32($afrt, $pos + 4);
          $qualityEntryCount = ReadByte($afrt, $pos + 8);
          $pos += 9;
          for ($i = 0; $i < $qualityEntryCount; $i++)
              $qualitySegmentUrlModifiers[$i] = ReadString($afrt, $pos);
          $fragEntries = ReadInt32($afrt, $pos);
          $pos += 4;
          LogDebug(sprintf(" %-12s%-16s%-16s%-16s", "Number", "Timestamp", "Duration", "Discontinuity"));
          for ($i = 0; $i segTable);
          $lastSegment   = end($this->segTable);
          $firstFragment = reset($this->fragTable);
          $lastFragment  = end($this->fragTable);

          // Check if live stream is still live
          if (($lastFragment['fragmentDuration'] == 0) and ($lastFragment['discontinuityIndicator'] == 0))
            {
              $this->live = false;
              array_pop($this->fragTable);
              $lastFragment = end($this->fragTable);
            }

          // Count total fragments by adding all entries in compactly coded segment table
          $invalidFragCount = false;
          $prev             = reset($this->segTable);
          $this->fragCount  = $prev['fragmentsPerSegment'];
          while ($current = next($this->segTable))
            {
              $this->fragCount += ($current['firstSegment'] - $prev['firstSegment'] - 1) * $prev['fragmentsPerSegment'];
              $this->fragCount += $current['fragmentsPerSegment'];
              $prev = $current;
            }
          if (!($this->fragCount & 0x80000000))
              $this->fragCount += $firstFragment['firstFragment'] - 1;
          if ($this->fragCount & 0x80000000)
            {
              $this->fragCount  = 0;
              $invalidFragCount = true;
            }
          if ($this->fragCount fragCount = $lastFragment['firstFragment

 

image

 

4.) You should now have a single .FLV video file with all the fragments joined together. If you use a media player that can play FLV video, such as VLC you can stop here. However, if you want to make the video a slightly more standard .MP4 format, you can convert it using FFMPEG, like this:

ffmpeg -i The_Music_Mash_Mustard_TV_video_1200_1080Seg1_Frag.flv -vcodec copy -acodec copy output.mp4
 
Posted in Uncategorized | Tagged , | Leave a comment

Underwater Effects

There’s lots of resources that describe techniques for realistic rendering of water surfaces. Unity comes supplied with several standard water assets, and there are some excellent descriptions of more advanced water effects at scrawkblog to generate spectacular ocean surfaces such as this:

Or Martin Upitis’ blender file: http://www.blendswap.com/blends/view/68857

However, I see much less information and examples of creating underwater effects, so here’s some notes on how I’ve recently approached different elements of making an underwater game scene.

 

a.) Fog

However clear the water in which you’re swimming, visibility is always going to be less than that in air, and global fog is a pretty easy way to achieve this. I chose a grey-blue colour (say, RGB 60,100,120) with Exp2 fog mode, and density of around 0.005 – adjustable depending on how murky you want it.

 

b.) Sea Floor

I got this nice diffuse tiling sand texture from CGTextures.

image

I like this image because instead of having a detailed grain texture, it has those nice smooth marks that you expect from sand that (at least at some point in time) has had waves wash over it. I then generated a normal map from the texture using the GIMP NormalMap plugin (not to be confused with the BumpMap plugin):

image

Then I created a simple Quad and applied a bumped diffuse material to it using these two textures. For more detail you could of course use a sculpted mesh or terrain, but a flat plane was fine for me to start with. Here it is, along with the underwater fog defined previously:

image

 

c.) Caustics

Caustics are the patterns of light you see projected onto surfaces under the water.

http://habibs.files.wordpress.com/2008/04/caustics_rendering_example.jpg?w=640

Caustics are easily simulated by using a projector to show an animated texture. The projector should be set to point straight down (i.e. rotation of 90 degrees about the X axis), with an orthographic projection. Then, attach the following script to the projector, which will allow it to cycle through a series of frames of animation:

using UnityEngine;
using System.Collections;

public class AnimatedProjector : MonoBehaviour
{
    public float fps = 30.0f;
    public Texture2D[] frames;
    private int frameIndex;
    private Projector projector;

    void Start()
    {
        projector = GetComponent();
        InvokeRepeating("NextFrame", 0, 1 / fps);
    }

    void NextFrame()
    {
        projector.material.SetTexture("_MainTex", frames[frameIndex]);
        frameIndex = (frameIndex + 1) % frames.Length;
    }
}

To create the frames of caustic animation, you can use a tool such as http://www.dualheights.se/caustics/ (free for non-commercial use). I find that you’ll need to generate somewhere between 30 – 60 frames to get a convincing animation. The frames should look something like this:

image

Assign these textures to the frames[] array of the AnimatedProjector script above, and use a shader with an additive blend mode so that the white parts of the texture lighten the surface onto which they are projected. The caustic generator above creates tileable images, so you can position a grid of projectors next to each other if you want to cover a large area (that’s why you should also use an orthographic projection, to ensure the images don’t get distorted at the edges, which would mess up the tiling).

 

d.) Bubbles

Although I’ve personally never seen anything on the seabed that generates a constant stream of bubbles, it seems to be standard fare in computer games for conveying a sense of being underwater.

http://critical-gaming.com/storage/coin6.4.png?__SQUARESPACE_CACHEVERSION=1259600392411

These are easy to achieve using a particle emitter to create billboard 2D sprites of a simple bubble texture, drawn using an additive shader. Here’s one example of a suitable bubble texture:

image

If you want to get fancy, you can create a slightly animated texture or apply random forces to make the bubbles “wobble” as they rise to the surface, but generally it’s acceptable to emit them at an angle close to vertical (here I use a cone emitter with radius of 5 degrees). You can add some slight randomness to the size and speed of bubbles too. I don’t understand the exact, complicated physics, but I believe bubbles grow larger as they rise towards the surface, and also large bubbles rise faster than small bubbles, which I’ve implemented through the “Size over lifetime” and “Size by speed” curves. Finally, I make my bubbles “fade out” using an alpha curve in “Color over lifetime”; I’m not sure this is realistic, but it prevents bubbles suddenly “popping” while still underwater. You could instead adjust the particle lifetime to ensure that bubbles just reach the surface before being destroyed. Here’s the complete settings I used:

image

and here’s the bubble stream they create:

image

 

e.) Flora, Fauna, and Furniture

The appropriate props to add to an underwater scene will depend on whether it’s intended to be tropical or polar, shallow or deep water, however, once again sticking with what people expect to find in an underwater game, I added a few rocks, corals, fish, and a sunken rowing boat. Goodness knows whether any of these are authentic or not, but at least they make the scene more interesting. You can find lots of suitable assets, many of them free, on the Unity Asset Store.

 

The Finished Scene

Here’s the final scene – nothing particularly innovative or hardware-taxing, and certainly suitable for mobile devices, but hopefully conveying what players expect from an underwater scene:

image

 

And a video to show the effect of the caustics in action…

Posted in Game Dev | Tagged , , , | Leave a comment

The Hunt for Napoleon

I was commissioned to create a game for the “Nelson’s Norfolk” exhibition that has been running in The Forum, Norwich throughout August. “The Hunt for Napoleon” is a first-person adventure set during the Battle of the Nile, in which players explore a ship and read the text of letters written by Nelson as he pursued the French fleet around the Mediterranean Ocean, revealing the British Admiral’s increasing frustration.

It’s always very exciting to see members of the public playing something you created (and occasionally, finding the bugs you didn’t spot!).

 BuhUjRqIgAA7sbG.jpg large

 

The game was created in Unity using a modified UFPS character controller for character movement. The main ship model was based on the “Sea Queen” asset by Mr Necturus. The text of the letters was taken from the archives of the War Times Journal. Everything else (including level design, scripting, shaders, GUI, music and SFX…) was created by me.

 

letter10325231_617682888349107_7318743040314865118_n10307210_617682771682452_8729974520189879512_n

 

I’m planning to write some followup posts about particular interesting challenges in creating a game for public display. There’s been lots of positive feedback, so I’m hoping to create further games for public exhibitions in the future (and hopefully learn from some of my own findings this time!).

Posted in Game Dev | Tagged , , , , | Leave a comment

Batch Conversion of BVH to FBX Motion Capture files

Continuing my foray into the world of motion capture, I’ve been building up my own collection of mocap data files from various sources I find on the ‘net, with the hope of ultimately using them to animate a Mecanim rig in Unity.  While I don’t think Mixamo will be worried quite yet(!), I have gathered enough BVH files such that loading them up individually into Blender and saving them to FBX necessary for import in Unity has become quite a chore.

So, I wrote a python script to automate the process, convert_fbx.py, as follows:

import bpy
import sys

#Get command line arguments
argv = sys.argv
argv = argv[argv.index("--") + 1:] # get all args after "—"
bvh_in = argv[0] 
fbx_out = argv[0] + ".fbx"

# Import the BVH file
# See http://www.blender.org/documentation/blender_python_api_2_60_0/bpy.ops.import_anim.html
bpy.ops.import_anim.bvh(filepath=bvh_in, filter_glob="*.bvh", global_scale=1, frame_start=1, use_fps_scale=False, use_cyclic=False, rotate_mode='NATIVE', axis_forward='-Z', axis_up='Y')

# Export as FBX 
# See http://www.blender.org/documentation/blender_python_api_2_62_1/bpy.ops.export_scene.html
bpy.ops.export_scene.fbx(filepath=fbx_out, axis_forward='-Z', axis_up='Y', use_anim=True, use_selection=True, use_default_take=False)

 

Then, create a batch file, bvh2fbx.bat, that loops through all BVH files in a directory and calls Blender from the command line with the script above, passing in the details of files to be converted:

 

FOR %%f IN (*.bvh) DO "c:\program files (x86)\blender foundation\blender\blender.exe" -b --python "c:\temp\convert_fbx.py" -- "%%f"

 

(Alter the path to the Blender executable and to where you saved the convert_fbx.py script as appropriate). Run the bvh2fbx.bat file from the commandline and you should get all BVH files in the directory automatically converted to FBX.

image

 

All that remains is to import them to Unity and add a Humanoid rig and they’re ready to use in an Animator!

image

Posted in Game Dev | Tagged , , , , , , | 2 Comments

A Simple Mind-Reading Machine

Just for fun, I decided to implement a "Mind Reading Machine" based on the work of Claude Shannon and D.W. Hagelbarger. If you’re not familiar with it, there’s a good summary of Shannon’s idea here (together with descriptions of some of his other inventions).

Essentially, it’s an algorithm that allows a computer to play a simple game against a human opponent. In each round of the game, the player has a free choice between two options – left/right, or heads/tails, for example. The algorithm attempts to "guess" which the player will choose. And, more often than not, the computer guesses correctly.

How does it work? Not by mind-reading, obviously, but by exploiting the fact that humans do not behave as "randomly" as they think they do. The computer maintains a very simple memory that records the pattern of results of the last two rounds – whether the player won or lost, whether they switched strategy, and then whether they then won or last the following round. The computer then uses this to choose its own best strategy based on the way the player behaved last time the current pattern occurred. If the computer loses twice in a row using the current strategy, it picks a random response in the next round.

There are ways to beat the algorithm (by ensuring that each time a pattern of plays comes up, you always behave the opposite way than you did last time), and I suspect there are many ways in which it could be improved (perhaps using Context-Tree Weighting, for example – but it’s a still an interesting experiment in its current form…

Here’s some C# code with a simple implementation of the algorithm – simply attach to an empty gameobject in a Unity scene (or the camera will do), and have a go at outwitting the computer by creating a "random" pattern using the left/right arrow keys:

using UnityEngine;
using System.Collections;

public class MindReader : MonoBehaviour {
     
    // Used to record play history
    int[,,] v;
    int lw1, lw2;
    int losingstreak;
    
    // The prediction of the player's next turn
    int prediction;
    
    // Keep track of scores
    int cpuScore, playerScore;

    string outText;
    
    // Use this for initialization
    void Start () {
        
        v = new int[2,2,2] { {{0,0},{0,0}}, {{0,0},{0,0}} };
        
        // No prior knowledge, so set initial prediction based on random coin flip
        prediction = flip ();
        
        // Set scores to 0
        cpuScore = playerScore = 0;
        
        // Won or lost last play and play before last
        lw2 = lw1 = 0;
        
         // Output
        outText = "";
        
    }
    
    void TakeTurn(int playerChoice) {

        // Display each player and computer's choices
        string outTextString = "You pressed " + playerChoice + ", " + "I guessed " + prediction + "\n";
         
        // Computer correctly guessed
        if(playerChoice == prediction) {
            cpuScore++;
            losingstreak = 0;
            outTextString += " I WIN!\n";
        }
         // Computer incorrectly guessed
        else {
            playerScore++;
            losingstreak++;
            outTextString += " YOU WIN!\n";
        }
        
        outText = outTextString;
         
        // Record latest scores in the matrix
        v[lw2,lw1,1] = (v[lw2,lw1,0] == playerChoice ? 1 : 0);
        v[lw2,lw1,0] = playerChoice;
        lw2 = lw1;
        lw1 = playerChoice;
        
        // If lost more than twice in present state, chose random strategy
        // Otherwise, keep same strategy
        prediction = v[lw2,lw1,1]==1 && losingstreak < 2 ? v[lw2,lw1,0] : flip();
    }
     
    // Simulate a coin flip to produce 50:50 chance of {0,1}
    int flip() {
        return Random.Range(0,2);
    }
    
    
    // Update is called once per frame
    void Update () {
    
         if(Input.GetKeyDown(KeyCode.LeftArrow)){
            TakeTurn(0);
        }
        if(Input.GetKeyDown(KeyCode.RightArrow)){
            TakeTurn(1);
        }    
    }
    
    void OnGUI() {
         GUIStyle style = new GUIStyle();
        style.fontSize = 36;
        
        GUI.Label(new Rect(0,0,Screen.width,100), outText, style);    
        GUI.Label(new Rect(0,100,Screen.width,200), "Player:" + playerScore + "CPU:" + cpuScore, style);
     }
    
}

There’s also a Java implementation of the same algorithm which you can try online at http://seed.ucsd.edu/~mindreader/

Posted in AI | Tagged , , | Leave a comment

Project Spark : First Impressions

Project Spark is an upcoming game/game creation environment/educational coding tool/interactive story creator/all/none of the above. Actually, I’m not quite sure what it is (nor exactly who it’s aimed at), but recently I’ve been doing a lot of computing with kids (both my own and others!) and, amongst the relative dirge of other latest-gen releases, it really stood out as something of interest to both me and them – it’s more pretty to look at than Scratch, say (which is excellent in many other respects), and it’s more educational and programming-like than other creative games like LittleBigPlanet or Minecraft.

It’s currently available only in private beta (due for release on Windows, XBox360, and XBoxOne sometime later this year I think), but I got offered a key so thought I’d take it up and give it a go.

You can play Project Spark games that others have created, but my interest was in creating my own project, so I started “Creative” mode. The tools to sculpt and paint your game world are pretty intuitive to use – my 6yr old had no difficulty dragging up mountains, creating rivers, grasslands, and placing trees – it’s essentially just a 3d painting program, but the results certainly do look pretty.

sculpt

However, what I was more interested in was the approach to creating game logic. The difficulty with any tool like this is getting the right balance of freedom –v- complexity. Oversimplify and, while you make the tool easy to use, it’s frustratingly limiting when trying to implement a feature not foreseen by the developers. Give the user more power and choice, however, and they become overwhelmed by complexity.

And that’s where Project Spark seems to be rather well-designed: there’s a gallery of game characters and other props that you can simply drag into your game world and, out-of-the-box, they pretty much behave as you’d expect in a game: “Hero” characters swing their sword with the X button on a gamepad, the camera can be moved with the mouse, doors can be opened by standing next to them and pressing space, goblins chase the hero, etc. etc. All without any additional effort.

But what’s smart is that all those behaviours only exist because that’s the default rules specified in those objects’ “brains”. Those brains can be edited: individual brain “tiles” can be swapped, rules added or deleted, or just completely rewritten from scratch. Here, for example is part of the brain that comes with the default player character:

brain

The code tiles are arranged to read pretty much like English: “When A pressed, do jump”, “When X pressed or left mouse button pressed, do attack”, “When an interactable object is detected in front, highlight it in yellow”. Changing or extending behaviour can be done by simply dragging alternative tiles into the brain.

Not only does that mean that it’s easy for novice programmers to create a brain that produces desired behaviour but, as a learning tool it also provides the reverse: having dropped a goblin (or, rather, a horde of them) into the game and witnessed their default behaviour, my son could go into their brains and see the set of rules that produced the behaviour in question. That, I think, is an incredibly powerful way to learn programming logic.

Technologically, and aesthetically, I’ve been very impressed, and if you get the chance to try Project Spark out for yourself, I’d highly recommend it. My only concern relates to its financing/distribution model:although this might be premature since the game hasn’t even been released yet, it appears that  the game itself will be free, and come with a base set of in-game objects from the gallery. Additional objects to make your games more varied and exciting can then be purchased through in-game microtransactions. Perhaps I’m just stuck in my ways, but I am not a fan of IAP, especially not in games that have such obvious appeal to kids as Project Spark does. I for one hope there will be an option just to pay £40 (heck, maybe £50) for the game up-front replete with a full complement of game objects and then let my children loose to create their own imagined game world, without fear of them being barraged by “Pay 1,000 credits to use this enchanted stick of awesome power?”-style messages…

Posted in Game Dev | Tagged , | Leave a comment

Glass Shader

I’ve seen a lot of posts recently from folks trying to create shaders that recreate the appearance of glass. I guess the reason is that glass is a pretty common material, but modelling its appearance is surprisingly complex – requiring transparency, reflection, specular highlights, texture maps, and more.

For what it’s worth, here’s my attempt at a transparent specular shader suitable for use in a glass material. It’s pretty self-explanatory I think:

Shader "Custom/Transparent Specular" {
	Properties {
		_MainTex ("Base (RGB)", 2D) = "white" {}
		
		// Colour property is used only to set influence of alpha, i.e. transparency
		_Colour ("Transparency (A only)", Color) = (0.5, 0.5, 0.5, 1)
		// Reflection map
		_Cube ("Reflection Cubemap", Cube) = "_Skybox" { TexGen CubeReflect }
		// Reflection Tint - leave as white to display reflection texture exactly as in cubemap
		_ReflectColour ("Reflection Colour", Color) = (1,1,1,0.5)
		// Reflection brightness
		_ReflectBrightness ("Reflection Brightness", Float) = 1.0
		// Specular
		_SpecularMap ("Specular / Reflection Map", 2D) = "white" {}
		// Rim
		_RimColour ("Rim Colour", Color) = (0.26,0.19,0.16,0.0)
	}
	SubShader {
		// This will be a transparent material
		Tags {
			"Queue"="Transparent"
			"IgnoreProjector"="True"
			"RenderType"="Transparent"
		}
		CGPROGRAM
		// Use surface shader with function called "surf"
		// Use the inbuilt BlinnPhong lighting model
		// Use alpha blending
		#pragma surface surf BlinnPhong alpha

		// Access the Shaderlab properties
		sampler2D _MainTex;
		sampler2D _SpecularMap;
		samplerCUBE _Cube;
		fixed4 _ReflectColour;
		fixed _ReflectBrightness;
		fixed4 _Colour;
		float4 _RimColour;

		struct Input {
			float2 uv_MainTex;
			float3 worldRefl; // Used for reflection map
			float3 viewDir; // Used for rim lighting
		};

		// Surface shader
		void surf (Input IN, inout SurfaceOutput o) {
		
			// Diffuse texture
			half4 c = tex2D (_MainTex, IN.uv_MainTex);
			o.Albedo = c.rgb;
			
			// How transparent is the surface?
			o.Alpha = _Colour.a * c.a;
			
			// Specular map
			half specular = tex2D(_SpecularMap, IN.uv_MainTex).r;
			o.Specular = specular;
			
			// Reflection map
			fixed4 reflcol = texCUBE (_Cube, IN.worldRefl);
			reflcol *= c.a;
			o.Emission = reflcol.rgb * _ReflectColour.rgb * _ReflectBrightness;
			o.Alpha = o.Alpha * _ReflectColour.a;
			
			//Rim
			// Rim lighting is emissive lighting based on angle between surface normal and view direction.
			// You get more reflection at glancing angles
			half intensity = 1.0 - saturate(dot (normalize(IN.viewDir), o.Normal));
			o.Emission += intensity * _RimColour.rgb;
		}
		ENDCG
	} 
	FallBack "Diffuse"
}

The Colour property is used only to set the overall transparency of the surface. The MainTex can be used to show grime, fingerprints, smudges, and smears on the surface. The SpecularMap should be set to match this by adjusting the degree to which specular highlights are shown. The Cube map contains the reflected environment image (which can be created using the script in my previous post). Here’s example textures for the maintex and spec map:

glass glass_s

 

When rendered in a scene, this gives a result as follows:

image

Posted in Game Dev | Tagged , | 1 Comment