The general form of the nearest neighbor query is: “Show me the closest x restaurants/pubs/train stations to this location”; this is the query pattern used by location-aware applications to find POIs close to the user’s current location, for example. However, another pattern of which I’ve seen much fewer examples is how to update an entire table of records in order to determine and populate the nearest neighbour for each row in the table (indeed, this very question came up on the MSDN spatial forum just last week).

With this in mind, I thought I’d demonstrate a practical scenario: let’s say you’re planning going for a swim at the local pool and, after you’ve swum 50 lengths, you’ll probably feel a bit peckish – perhaps in the need for some fish ‘n’ chips. So, in this example I’m firstly going to import the Garmin POI data for swimming pools and fish ‘n’ chip shops in the UK, each being represented by a geography Point instance. I’ll then add an extra column to the swimming pool table and populate it with the closest fish and chip shop to every swimming pool.

To start with, create tables containing the data for fish and chip shops and swimming pools (download the data from here). Notice that I’ve included an IDENTITY primary key field in each table, which I’ll need in order to add a spatial index later:

SELECT
  Id = IDENTITY(int,1,1),
  CAST(Name AS varchar(255)) AS Name,
  CAST(Details AS varchar(255)) AS Details,
  geography::Point(Latitude,Longitude,4326) AS Location
INTO SwimmingPools
FROM OPENROWSET(
 BULK 'C:\Users\Alastair\Downloads\GarminPOI\Swimming Pools.csv',
 FORMATFILE='C:\temp\garminpoi.xml',
 FIRSTROW=2
) AS SwimmingPools;
GO

ALTER TABLE SwimmingPools ADD PRIMARY KEY(Id);
GO

SELECT
  Id = IDENTITY(int,1,1),
  CAST(Name AS varchar(255)) AS Name,
  CAST(Details AS varchar(255)) AS Details,
  geography::Point(Latitude,Longitude,4326) AS Location
INTO FishAndChips
FROM OPENROWSET(
 BULK 'C:\Users\Alastair\Downloads\GarminPOI\Fish and Chips.csv',
 FORMATFILE='C:\temp\garminpoi.xml',
 FIRSTROW=2
) AS FishAndChips;
GO

ALTER TABLE FishAndChips ADD PRIMARY KEY(Id);
GO

Now we’ll add some extra columns to the Swimming Pools table, which we’ll populate with information about the closest fish ‘n’ chip shop:

ALTER TABLE SwimmingPools
ADD ClosestFishbar varchar(255),
    Distance decimal(18,2);

Now, you don’t need to add a spatial index to perform the update query, but it’ll be a lot faster with one than without. SQL Server can only make use of one spatial index in a join between tables, so there’s no need to add an index to both tables. We’ll just add one to the Fish And Chips table, as follows:

CREATE SPATIAL INDEX sidx_FishAndChips ON FishAndChips(Location)
USING  GEOGRAPHY_GRID 
WITH (
  GRIDS =(LEVEL_1 = HIGH,LEVEL_2 = HIGH,LEVEL_3 = HIGH,LEVEL_4 = HIGH), 
  CELLS_PER_OBJECT = 16
);

Then, run an UPDATE query to populate the ClosestFishBar and Distance columns, as follows:

UPDATE s
SET ClosestFishbar = fName,
    Distance = Location.STDistance(fLocation)  
FROM (
  SELECT
    SwimmingPools.*,
    fnc.Name AS fName,
    fnc.Location AS fLocation
  FROM SwimmingPools 
  CROSS APPLY (
    SELECT TOP 1
     Name,
     Location
    FROM FishAndChips WITH(index(sidx_FishAndChips))
    WHERE FishAndChips.Location.STDistance(SwimmingPools.Location) IS NOT NULL
    ORDER BY FishAndChips.Location.STDistance(SwimmingPools.Location) ASC
  ) fnc
) s;

This query probably needs some explanation:

• Generally, you might expect to use a correlated subquery to obtain the closest fish and chip shop to each swimming pool. However, correlated subqueries can only return a single value, and I want to retrieve both the name and the distance to the closest fish bar, so I’ve used a CROSS APPLY instead.
• In the function being applied, I’m sorting the table of fish and chip shops by ascending order of their distance from the current swimming pool (ORDER BY FishAndChips.Location.STDistance(SwimmingPools.Location) ASC), and then selecting the TOP 1 Name and Location (i.e. the closest).
• To make the query efficient, I’ve also added an index hint to use the spatial index on the Fish and Chips table – WITH(index(sidx_FishAndChips)), and I’ve included an extra predicate, WHERE FishAndChips.Location.STDistance(SwimmingPools.Location) IS NOT NULL. This extra predicate will help the query optimiser choose the dedicated nearest neighbour query plan introduced in SQL Server 2012 and SQL Azure.
• Note that the spatial index-optimised nearest neighbour query plan is only available in SQL Server 2012/SQL Azure. If you try to execute the UPDATE query as written above in SQL Server 2008/R2 then you’ll get an error:

  “The query processor could not produce a query plan for a query with a spatial index hint.  Reason: Spatial indexes do not support the comparator supplied in the predicate.  Try removing the index hints or removing SET FORCEPLAN.”

  As stated, to resolve this you’ll have to remove the WITH(index(sidx_FishAndChips)) index hint (and also expect your query to run a lot slower!). Instead, you might need to try one of the alternative approaches to efficiently finding nearest neighbours in SQL Server 2008/R2.

• Finally, in order to perform the UPDATE, I’ve wrapped the whole lot in a table alias, s, and then updated the two columns as required.

This query should take a few seconds to run (or, if you don’t create/use the spatial index, a few minutes), after which you can check the results as follows:

SELECT * FROM SwimmingPools;

And there you have the complete table of swimming pools, each listed with the name and distance to its closest fish and chip shop:

Seeing as it’s less than 100 metres from getting out of the water to a bag of chips, I’ll think I’ll be taking my next dip at the Farnworth swimming pool….

SET @h = geometry::STGeomFromText('POINT(10 10)', 0);

However, people often tell me that they really appreciate real-world examples instead – not only are they more interesting, but it’s easier to understand a practical use for the method, and they’re also more memorable afterwards. Some examples I’ve used in the past include:

• Use STBuffer() to calculate the free delivery area around a Pizza restaurant
• Use STDisjoint() to ensure that a planned road development does not intersect a protected area of natural beauty
• Use STIntersection() to identify the section of the Via Appia Roman road that passed through the malaria-infested Pontine marshes.

Anyway, I thought I’d share a new source of data I came across recently – it’s a free dataset of (UK) points of interest designed for importing into Garmin sat-nav systems. As such, it contains practical POIs, categorised into things like museums, mountains, petrol stations, and restaurants. However, since it’s distributed in a simple CSV format, it’s super easy to import the data into SQL Server or other systems as well. Here’s how:

1. Download

Download the dataset of POIs from the garminpoi.co.uk website at http://garminpoi.co.uk/?page=download

When unzipped, you’ll find it contains a number of categorised CSV files (each with an accompanying BMP icon, which you could use if you want to display this data using pushpins on an SSRS report or Bing Maps, for example):

2. Create a Format File

All of the CSV files follow exactly the same structure – with columns for Longitude, Latitude, a Name, and Description. This can be described by the following bulk format file, which you should save as garminpoi.xml:

<?xml version="1.0"?>
<BCPFORMAT xmlns="http://schemas.microsoft.com/sqlserver/2004/bulkload/format" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<RECORD>
 <FIELD ID="0" xsi:type="CharTerm" TERMINATOR=","/>
 <FIELD ID="1" xsi:type="CharTerm" TERMINATOR=","/>
 <FIELD ID="2" xsi:type="CharTerm" TERMINATOR=","/>
 <FIELD ID="3" xsi:type="CharTerm" TERMINATOR="\r\n"/>
</RECORD>
<ROW>
 <COLUMN SOURCE="0" NAME="Longitude" xsi:type="SQLVARYCHAR"/>
 <COLUMN SOURCE="1" NAME="Latitude" xsi:type="SQLVARYCHAR"/>
 <COLUMN SOURCE="2" NAME="Name" xsi:type="SQLVARYCHAR"/>
 <COLUMN SOURCE="3" NAME="Details" xsi:type="SQLVARYCHAR"/>
</ROW>
</BCPFORMAT>

3. Open the CSV file in SQL Server

Now, you can access the data from the CSV file directly in SQL Server using OPENROWSET, supplying the name of the CSV file you want to load following the BULK keyword, and the formatfile created above following the FORMATFILE keyword. The CSV files contain column headers, which you can skip by specifying FIRSTROW=2. What’s more, you can pass the Latitude and Longitude coordinate values from the CSV straight to the Point() method of the geography datatype to create point instance at each location on-the-fly.

For example, try executing the following code listing:

SELECT *,
geography::Point(Latitude,Longitude,4326) AS TescoStore
FROM OPENROWSET(
 BULK 'C:\Users\Alastair\Downloads\GarminPOI\Tesco.csv',
 FORMATFILE='C:\temp\garminpoi.xml',
 FIRSTROW=2
) AS GarminPOI;

Result? An instant map of all Tesco branches in the UK!

Unfortunately, as is often the case with OS software, it seems that almost all Mapnik developers use PostGIS as a backend spatial database, and I failed to find anybody with any experience of SQL Server on either the Mapnik developers mailing list or on StackExchange (rather amusingly, the top-rated answer given to me on StackExchange gives, as its reference source, my own blog post where I explicitly state that the suggested approach doesn’t work!).

Since, at the time, I needed a tool to render SQL geometry data there and then, I left Mapnik alone and developed a solution using MapServer instead.

Now, MapServer is a great product: It’s a venerable and highly-respected product in the world of open source spatial software, developed by the University of Minnesota on behalf of NASA, and first released over 16 years ago (to put that in context: MapServer was released at around the same time as HTML 3.2 – long before CSS, AJAX, or Google Maps were heard of). And it’s still going strong today. However, I find that the archaic MapServer architecture and idiosyncracies of its syntax give away its age slightly.

Mapnik, by contrast, is freshly-designed, has a modern XML-style syntax, supports groovy things like node.js and Ruby, and generally feels like it’s not carrying so much baggage. If only it supported SQL Server….

6 months later, and the Release Candidate for Mapnik 2.0 is  now available to download, which claims to support SQL Server, so I thought I’d give it another try. Having just set it up and rendered some quick OS OpenData, I can confirm that it works. Hurray!

SSMS Spatial Results Tab Rendering of SQL Server data

Mapnik Rendering of the same SQL Server data

Since it seems that I’m one of only few people trying to use Mapnik and SQL Server together (and one of even less writing about it), I’ve started to document some tips in a page here – I’ll add to this page as I go along.

(On a separate note, I now have a whole different problem, because a recent change in the Bing Maps Terms of Use prevents you from adding your own custom road tile layers onto the Bing Map control. So, any tiles of OpenStreetMap or Ordnance Survey road data you create with Mapnik can’t be used with Bing Maps. Fortunately, there’s a workaround for that, but that’s for another post….)

Msg 6522, Level 16, State 1, Line 1
A .NET Framework error occurred during execution of
user-defined routine or aggregate "geography":

Microsoft.SqlServer.Types.GLArgumentException: 24205:
The specified input does not represent a valid geography instance
because it exceeds a single hemisphere. Each geography instance
must fit inside a single hemisphere. A common reason for this
error is that a polygon has the wrong ring orientation.

This error arises out of a widely documented limitation in SQL Server 2008/R2 that geography instances cannot span more than a single hemisphere (i.e. they cannot extend over more than half the surface of the earth).

While, strictly speaking, this is a technical limitation, the “smaller-than-a-hemisphere” requirement actually has a beneficial side-effect of providing a useful sanity check of your data: in practice, it is very unusual to ever need a geography instance that spans more than a single hemisphere. Unless you are modelling the Pacific Ocean, or the entire continent of Asia as a single instance, the chances are that you didn’t really want a geography bigger than a hemisphere anyway. Instead, as the error message above states, what probably happened is that the coordinate data of your imported shape had the incorrect ring orientation – the coordinate points were listed in reverse order, which has the effect of inverting the polygon. Therefore, you ended up including everything that was meant to be excluded, and excluding the area that you meant to include. Instead of defining the county of Norfolk, for example, your polygon actually represents the entire surface of the earth except for Norfolk…

Assuming that what you really wanted was to define a relatively small polygon area – a country, state, or sales region, say, then the “within a hemisphere rule” ensures the quality of your data by rejecting any Polygon instances with the incorrect ring orientation.

Along Comes SQL Server 2012

Well, SQL Server 2012 now allows geography Polygons to be bigger than a hemisphere, and that brings a whole new interesting problem… because now, you no longer get an error when you try to import data with incorrect ring orientation – it loads just fine. And you might not even realise that there’s anything wrong with your data until you try to perform some calculations on it.

For example, I just downloaded some Australian river basins data from the Geoscience Australia website, and imported it into SQL Server using OGR2OGR. Everything seemed to go successfully, and here’s a glance at what the table looks like with the geography data inserted into the geog4202 column:

Seems ok, right? But if you calculate the area of any of the river basins in this table using STArea(), you’ll get a result like this:

SELECT TOP 1 geog4202.STArea();

-- 510,069,272,680,889 (m2)

I mean, I know Australia is a big place, but… a river basin with an area of 510,069,272 km²… Really? That seems awfully close to the entire surface area of the earth. And, if you look on the spatial results tab, you’ll see this:

Sure enough – it looks pretty much like those polygons do cover the entire surface area of the earth.

The problem here is that the source data does not follow the same “left-hand rule” convention as SQL Server, and some (the majority, in fact) of the imported river basin polygons have been defined inside-out. But, because SQL Server now allows large geographies, no error or warning has been created.

The Solution

Fortunately, SQL Server 2012, whilst inadvertently causing this problem, also provides the solution, in the form of the ReorientObject() method, which reverses the ring orientation of any polygon to flip its interior and exterior.

We only want to use ReorientObject() to flip those polygons that have been defined inside-out. Unfortunately, there’s no fool-proof way to check for this, but it’s reasonable to assume that, in a dataset of Australia’s river basins, any polygon that covers more than half the earth’s surface must be incorrect. So, we can identify those polygons that need flipping because their EnvelopeAngle will be equal to or more than 90°, and put this in a CASE statement as follows:

SELECT
  CASE
    WHEN geog4202.EnvelopeAngle() >= 90 THEN geog4202.ReorientObject()
    ELSE geog4202
  END AS geog4202
FROM rbasin_polygon;

And now, the results look much more like those expected:

(If you really want, you could even put a CHECK constraint on a geography column that prevented any polygons greater than a hemisphere being inserted into the table in the first place. Effectively, you’d be manually reintroducing a technical limitation from previous versions of SQL Server – ironic, huh?)

To investigate this puzzle using SQL Server, we’ll model a portion of the city  using two tables – one containing Polygons representing the landmasses separated by rivers, and another containing LineStrings representing the bridges across those rivers:

CREATE TABLE #islands 
(
islandid char(1),
islandgeom geometry
);

INSERT INTO #islands VALUES
('A', 'POLYGON((20.505 54.706, 20.506 54.708, 20.5077 54.7086, 20.513 54.7074, 20.514 54.7075, 20.52 54.7077, 20.52 54.71, 20.503 54.71, 20.503 54.706, 20.505 54.706))'),
('B', 'POLYGON((20.506 54.706, 20.5073 54.7083, 20.5135 54.7068, 20.5134 54.7053, 20.506 54.706))'),
('C', 'POLYGON((20.52 54.698, 20.5149 54.6999, 20.5137 54.7009, 20.51435 54.704, 20.5135 54.7046, 20.5051 54.7054, 20.503 54.7056, 20.503 54.698, 20.52 54.698))'),
('D', 'POLYGON((20.52 54.707, 20.5162 54.707, 20.514 54.707, 20.5142 54.7051, 20.515 54.704, 20.515 54.703, 20.5149 54.7008, 20.5159 54.7, 20.52 54.699, 20.52 54.707))');

CREATE TABLE #bridges(
bridgeid char(1),
bridgegeom geometry
);
INSERT INTO #bridges VALUES
(1, 'LINESTRING(20.5082 54.7092, 20.5075 54.7076)'),
(2, 'LINESTRING(20.5109 54.7087, 20.5101 54.707)'),
(3, 'LINESTRING(20.5148 54.708, 20.5147 54.70634)'),
(4, 'LINESTRING(20.5128 54.706, 20.5147 54.7057)'),
(5, 'LINESTRING(20.5069 54.70645, 20.5061 54.7048)'),
(6, 'LINESTRING(20.5103 54.706, 20.5099 54.70416)'),
(7, 'LINESTRING(20.5174 54.6985, 20.5187 54.7003)');

Each of the islands is lettered from A – D, and each bridge is numbered from 1 – 7, as shown in the results of the following query (I’ve applied a small buffer to the bridges only to make them easier to see in the spatial results tab):

SELECT islandid, islandgeom FROM #islands
UNION ALL
SELECT bridgeid, bridgegeom.STBuffer(0.0001) FROM #bridges

How, then, can we try to devise a route that crosses each bridge only once? When investigating this problem, the first conclusion reached by Euler was that the shape of the islands, and the path taken within any island, was irrelevant. All that matters is the possible connections that can be made between islands. (This conclusion, and Euler’s later work on the puzzle, led to the foundation of graph theory, in which the entities in a network can be modelled as nodes, and the relationships between those nodes defined by edges).

So, let’s create a table that maps every possible crossing between islands, the edges of the network, by using STIntersects() to determine those islands that are connected to each bridge:

CREATE TABLE #crossings(
  fromid char(1),
  viabridge char(1),
  toid char(1)
);

INSERT INTO #crossings
SELECT 
  f.islandid AS fromid,
  b.bridgeid AS viabridge,
  t.islandid AS toid
FROM #islands f
  JOIN #bridges b ON f.islandgeom.STIntersects(b.bridgegeom) = 1
  JOIN #islands t ON b.bridgegeom.STIntersects(t.islandgeom) = 1
WHERE f.islandid != t.islandid
ORDER BY f.islandid;

Having created a table with every individual bridge crossing, we can now attempt to create a continuous route across the city by navigating through that edges table using a recursive CTE.  The anchor member of this query starts by looking at each individual bridge crossing from the #crossings table. Having crossed a bridge to a new island,  the recursive member then joins back to the #crossings table to search for any bridges that begin on the island where the last crossing ended, in an attempt to continue the route (note the join condition – #crossings JOIN cte ON cte.toid = #crossings.fromid).

Along the way, the query concatenates the ids of all the bridges crossed into the bridgescrossed field. This field is used in a condition of the recursive member to ensure we don’t cross any bridge that’s already been included at some point in the route (WHERE bridgescrossed NOT LIKE ‘%’ + #crossings.viabridge + ‘%’).

Finally, we also concatenate both the bridge identifier and the islands visited in turn to create a nice formatted route of the journey for later examination. e.g. a journey that started at island A and went to island D via bridge 3 would be shown as A – 3 – D.

Here’s the full code listing:

WITH cte as (
  SELECT
    #crossings.fromid,
    CAST(#crossings.viabridge AS varchar(32)) AS bridgescrossed,
    CAST(#crossings.fromid + '-' + #crossings.viabridge + '-' + #crossings.toid AS varchar(32)) as formattedroute,
    #crossings.toid
    FROM #crossings
  UNION ALL SELECT
    #crossings.fromid,
    CAST(cte.bridgescrossed + #crossings.viabridge AS varchar(32)) AS bridgescrossed,
    CAST(cte.formattedroute + '-' + #crossings.viabridge + '-' + #crossings.toid AS varchar(32)) as formattedroute,
    #crossings.toid
    FROM #crossings JOIN cte
    ON cte.toid = #crossings.fromid
    WHERE bridgescrossed NOT LIKE '%' + #crossings.viabridge + '%'
)
SELECT
  formattedroute,
  len(bridgescrossed) AS numbridgescrossed
FROM cte
ORDER BY len(bridgescrossed) DESC;

The results show every possible route across the city that never cross the same bridge more than once, ordered to show longest routes (i.e. those that cross the most bridges) first. If any route existed that crossed every bridge, we’d expect it to be listed at the top of the results, and for the numbridgescrossed value to be 7.

However, we find that no such route exists. The longest route(s) in the resultset crosses only 6 of the 7 bridges. One such route, starting at island D and ending at island A, is as follows:

D – 7 – C – 6 – B – 4 – D – 3 – A – 2 – B – 1 – A

And that is empirical proof, using SQL Server, that no solution to this problem exists.

As for understanding why no solution exists? Consider the number of bridges connected to each island. Bear in mind that every time you arrive on an island via a bridge, you must leave that island via a different bridge. It therefore follows that, in order to follow a continuous route that crosses every bridge,  every island (with the exception of the start and end island), must have an even number of bridges connected to it – a matched set of entry/exit bridge pairs. In the Konigsberg puzzle, however, Island B has 5 bridges connected to it, and Islands A, C, and D each have 3 bridges connected to them. Since, at most, only two of these islands can be the start/end points of the route, no solution is possible.

There seem to be several variations on this puzzle, but the version I know is as follows:

“A hunter starts at a location. He walks one mile south, one mile east, and then one mile north. He ends up back where he started, and sees a bear. What colour is the bear?”

The Solution

Before worrying about the colour of the bear, we need to resolve the apparent paradox in the positioning of the hunter. He has moved one mile south, one mile east, and one mile north, and ended up back where he started. Where could he have been in order for this to be true?

The most commonly-known answer is that the hunter started at the North Pole, the bear is therefore a polar bear and is white (don’t let it worry you that there are no polar bears at the north pole!). Let’s test whether this answer fits the criteria using SQL Server’s geography datatype.

Firsty, we define the hunter’s start point, @p0, at the North Pole. The latitude of the North Pole is 90 degrees. All lines of longitude converge at the North Pole, so you can use any longitude value. I’ll use an arbitrary longitude value of 0:

DECLARE @p0 geography = geography::Point(90, 0, 4326);

We now need to declare the first point the hunter travels to, @p1, which is one mile due South of the start.

• Travelling south reduces latitude, so @p1.Lat < @p0.Lat.
• Travelling due south involves maintaining a constant longitude, so @p1.Long = @p0.Long.
• The coordinates in this example use SRID 4326, so the results of any linear calculations are stated in metres. 1 mile = 1,609 metres (approx.), so we need @p0.STDistance(@p1) = 1609.
• Solving these criteria leads to the following point:

DECLARE @p1 geography = geography::Point(89.98559, 0, 4326);

The next point that the hunter travels to lies one mile due east of @p1. Therefore, it must lie on the same latitude as @p1, but have a greater longitude (assuming that the antimeridian is not crossed). Using the same logic for the previous point, we can deduce that this point lies at:

DECLARE @p2 geography = geography::Point(89.98559, 60, 4326);

Finally, we need to establish the final point, @pf, which must lie one mile due north of the last point, as follows:

DECLARE @pf geography = geography::Point(90, 60, 4326);

To meet the final criteria of the puzzle, the final point must be the same as the point at which the hunter stated, so @pf = @p0.

We can prove that the points described above meet the criteria for the puzzle as follows:

DECLARE @p0 geography = geography::Point(90, 0, 4326);
DECLARE @p1 geography = geography::Point(89.98559, 0, 4326);
DECLARE @p2 geography = geography::Point(89.98559, 60, 4326);
DECLARE @pf geography = geography::Point(90, 60, 4326);

IF @p1.Lat  @p1.Long
AND CAST(@p2.STDistance(@p1) AS INT) = 1609
SELECT '@p2 lies one mile due east of @p1';

IF @pf.Lat > @p2.Lat
AND @pf.Long = @p2.Long
AND CAST(@pf.STDistance(@p2) AS INT) = 1609
SELECT '@pf lies one mile due north of @p2';

IF @pf.STEquals(@p0) = 1
SELECT '@pf is the same as @p0';

This proves that the route p0, p1, p2, pf meets all the criteria described in the riddle and, using the methods of the geography datatype, we have proven that the North Pole is a valid solution to this puzzle. However, it is not the only solution – there are, in fact, an infinite number of locations on the earth’s surface that would meet the criteria of the puzzle…

…consider the second step of the hunter’s journey – walking due east along a parallel of the earth – a line of constant latitude. As you approach the poles, the circumference of the parallels of the earth become increasingly smaller. Starting at a point just north of the south pole, it is possible to walk east right around the earth along a circle that is one mile in circumference. This occurs at a latitude of about -89.9977, as can be shown by the following:

DECLARE @linestring geography = geography::STLineFromText('
LINESTRING(0 -89.9977, 15 -89.9977, 30 -89.9977, 45 -89.9977, 60 -89.9977, 75 -89.9977, 90 -89.9977, 105 -89.9977, 115 -89.9977, 120 -89.9977, 135 -89.9977, 150 -89.9977, 165 -89.9977, 180 -89.9977, 195 -89.9977, 200 -89.9977, 210 -89.9977, 225 -89.9977, 240 -89.9977, 255 -89.9977, 270 -89.9977, 285 -89.9977, 300 -89.9977, 315 -89.9977, 330 -89.9977, 345 -89.9977, 360 -89.9977)
', 4326);

SELECT 
@linestring,
@linestring.STLength();

This linestring is 1,609 metres long – one mile, but follows a line of constant latitude right around the earth. So, if the hunter starts at any point that is one mile north of this line, they would first walk one mile south onto the line, then one mile east along the line (taking them once around the earth), and finally one mile north again to return to their original starting point.

The following code illustrates an example of just one such point that lies one mile north of the line:

DECLARE @q0 geography = geography::Point(-89.98329, 0, 4326);
SELECT @q0.STDistance(@linestring);

Therefore, an alternative solution to the puzzle could be as follows:

DECLARE @q0 geography = geography::Point(-89.98329, 0, 4326);
DECLARE @q1 geography = geography::Point(-89.9977, 0, 4326);
DECLARE @q2 geography = geography::Point(-89.9977, 360, 4326);
DECLARE @qf geography = geography::Point(-89.98329, 360, 4326);

It’s harder to prove this solution with SQL Server because, since walking east along the line between @q1 and @q2 takes you on an entire revolution of the earth, the result reported by STDistance() for the distance between  the points will always be zero (rather than the 1,609 metres it actually took you to walk there).

In fact, not only are there an infinite number of points that lie one mile north of the line at constant latitude –89.9977 that satisfy the puzzle, but there are also an infinite number of lines along which it can be done: Consider the line that goes around the South pole that is 1/2 mile in length, for example, or 1/3 mile. Walking due east for one mile along any of these lines will take you on some number of rotations of the earth but always return you to the starting point. In all these cases, the middle step of “walking due east for one mile” effectively becomes nullified, leaving the puzzle as walking due south for one mile and then due north for one mile, which clearly returns you to the start location.

DECLARE @table1 TABLE( id CHAR(1), shape geometry );
INSERT INTO @table1 VALUES
('A','POLYGON((0 0, 8 0, 8 3, 0 0))'),
('B','POLYGON((8 0, 13 0, 13 2, 10 2, 10 1, 8 1, 8 0))'),
('C','POLYGON((8 3, 8 1, 10 1, 10 2, 13 2, 13 3, 8 3))'),
('D','POLYGON((8 3, 13 3, 13 5, 8 3))');

If you now select all of the shapes from the table and switch to the spatial results tab in SQL Server Management Studio, you can see that the individual shapes fit together to form a triangle:

SELECT id, shape
FROM @table1;

Now suppose that we insert these same shapes into another table, but move them around slightly:

DECLARE @table2 TABLE ( id CHAR(1), shape geometry );

INSERT INTO @table2 VALUES
('A','POLYGON((5 2, 13 2, 13 5, 5 2))'),
('B','POLYGON((8 0, 13 0, 13 2, 10 2, 10 1, 8 1, 8 0))'),
('C','POLYGON((5 2, 5 0, 7 0, 7 1, 10 1, 10 2, 5 2))'),
('D','POLYGON((0 0, 5 0, 5 2, 0 0))');

SELECT id, shape
FROM @table2;

Note that all of these shapes are identical to those used in the first example – they’ve just been rearranged. However, the triangle formed when the pieces are fitted together now contains a “hole” – 1 unit wide by 1 unit high.

So, where has the hole appeared from, and what is missing?

The Solution

The area of a right-angled triangle can be calculated as half of the base multiplied by the height. Both ‘triangles’ appear to be 13 units wide x 5 units high, so we would expect them to have an area of 32.5 units. Let’s test this:

SELECT SUM(shape.STArea()) FROM @table1;
SELECT SUM(shape.STArea()) FROM @table2;

The result of the above query shows that the total area of the individual shapes contained in both triangles is the same – 32 units, not 32.5 as expected. The area of the bottom triangle, when including the ‘missing’ one unit square, is therefore 33 units.

In other words, although the two triangles appear similar, they are not the same, and neither one is the 13×5 right-angled triangle they appear to be. The first triangle is slightly smaller, and the second triangle is slightly larger.

To investigate further, we can find the area of difference between these triangles and the “true” 13 x 5 right-angled triangle defined as follows:

DECLARE @truetriangle geometry = geometry::STPolyFromText('POLYGON((0 0, 13 0, 13 5, 0 0))', 0);

First, we need to create a union aggregate of all the component shapes that form each triangle, and then compare this to the true triangle using the STSymDifference() method (note I also use the STConvexHull() method on the second triangle in order to fill in the missing ‘hole’). If you’re using SQL 2012, you can use the UnionAggregate() method here but, if not, you can create a simple union aggregate using STUnion() as shown below:

-- Create union of all individual shapes in first triangle
DECLARE @triangle1 geometry = 'GEOMETRYCOLLECTION EMPTY';
SELECT @triangle1 = @triangle1.STUnion(shape) FROM @table1; 
 
-- Create union of all individual shapes in second triangle (filling in the gap)
DECLARE @triangle2 geometry = 'GEOMETRYCOLLECTION EMPTY';
SELECT @triangle2 = @triangle2.STUnion(shape).STConvexHull() FROM @table2; 
 
-- Work out the difference between each complete triangle and the true 13 x 5 triangle

SELECT
  @triangle1.STDifference(@truetriangle),
  @triangle1.STDifference(@truetriangle).STArea()
UNION ALL SELECT
  @triangle2.STDifference(@truetriangle),
  @triangle2.STDifference(@truetriangle).STArea();

The results illustrate two thin slivers of area where the two shapes differ from the true triangle, lying along the hypotenuse – each 0.5 units in area.

While both shapes formed from the component elements A, B, C, and D appear to be right-angled triangles, they are in fact polygons, since the hypotenuse of each is not a straight line. (You can check this by calling STNumPoints() on both @triangle1 and @triangle2 – you’ll see that each contains four distinct points). The hypotenuse of the @triangle1 is slightly concave, hence why the area is only 32 units and not 32.5 units as expected. The hypotenuse of the @triangle2 is slightly convex, leading to a total area of 33 units. When you add in the missing sliver to @triangle1, or take it away from @triangle2, you’ll end up with the true 13×5 triangle as expected.

The illusion is caused by the fact that the gradients of the component triangles A and D are not the same, therefore swapping them over affects the overall area of the shape. However, they are similar enough that your brain doesn’t perceive the difference between them, and assumes that you’re looking at a right-angled triangle in both cases.

The last few months have presented a number of changes for me, both personally and professionally speaking. In order to deal with them I had to enter a sort of “hibernation mode”, shutting down all non-essential activities, which includes blogging. (FWIW, I also have not been replying to many emails, and shut down my Twitter account).

Hopefully, things have started to calm down now, and I can get back to a (revised) normality again, which means this blog should start up again.

There’s lots of things to write about in the coming weeks – I’ve fallen in love with Mapnik again (but have fallen out somewhat with Bing Maps). SQL Server 2012 is nearly here, and I’ll be sharing some of the content from my new book. I’ve also got some new stuff about using spatial with SQL Azure, more OGR/GDAL goodness, and some real-life example applications I’ve been working on recently, including a nice spatially-enabled dashboard in SQL Server Reporting Services. As usual, if there’s anything you’d like me to write about, just let me know, and stay tuned for some new content soon!

What I’m not sure, though, is how to interpret this statistic or best act upon it. Does it mean, for example, that my articles about OGR2OGR – the open source toolkit for spatial data conversion and manipulation – are helpful, or high quality? Sure, several of them get listed on the first page of results for a Google search, but does that simply mean that nobody else on the internet is writing about OGR2OGR? And do the people that come here searching for information about OGR2OGR actually find what they’re looking for, or do they end up leaving empty-handed?

I don’t know the answers to these questions, but since there are obviously some people out there looking for this kind of information (and since I know, from experience, that OGR2OGR can be an absolute bugger to get working correctly) here’s another post about OGR2OGR anyway… and this time it’s about exporting spatial data from SQL Server. There’s plenty of information on the ‘net describing how to import spatial data from ESRI shapefiles into SQL Server (including my own posts, such as here, here and here, for example). However, what I haven’t seen any examples of yet is how to do the reverse: taking geometry or geography data from SQL Server and dumping it into an ESRI shapefile.

There’s plenty of reasons why you might want to do this – despite its age and relative limitations (such as the maximum filesize per file, limited fieldname length, the fact that each file can contain only a single homogenous type of geometry, etc…), the ESRI shapefile format is still largely the industry standard and pretty much universally read by all spatial applications. Recently, I needed to export some data from SQL Server to shapefile format so that it could be rendered as a map in mapnik, for example (which, sadly, still can’t directly connect to a MS SQL Server database).

So, here’s a step-by-step guide, including the pitfalls to avoid along the way.

Setup

To start with, make sure you’ve got a copy of OGR2OGR version 1.8 or greater (earlier versions do not have the MSSQL driver installed). You can either build it from source supplied on the GDAL page or, for convenience, download and install pre-compiled windows binaries supplied as part of the OSgeo4W package.

Now, let’s set up some test data in a SQL Server table that we want to export. To test the full range of OGR2OGR features (or should that say, “the full range of error messages you can create”?), I’m going to create a table that contains two different geometry columns – an original geometry and a buffered geometry, populated using a range of different geometry types:

CREATE TABLE OGRExportTestTable (
  shapeid int identity(1,1),
  shapename varchar(32),
  shapegeom geometry,
  bufferedshape AS shapegeom.STBuffer(1),
  bufferedshapearea AS shapegeom.STBuffer(1).STArea()
);

INSERT INTO OGRExportTestTable (shapename, shapegeom) VALUES
('Point #1', geometry::STGeomFromText('POINT(13 10)', 2199)),
('Point #2', geometry::STGeomFromText('POINT(7 12)', 2199)),
('Line #1', geometry::STGeomFromText('LINESTRING(0 0, 8 4)', 2199)),
('Polygon #1', geometry::STGeomFromText('POLYGON((2 2, 4 2, 4 4, 2 4, 2 2))', 2199)),
('Line #2', geometry::STGeomFromText('LINESTRING(0 10, 10 10)', 2199));

Here’s what the contents of this table looks like in the SSMS Spatial Results tab:

Exporting from SQL Server to Shapefile with OGR2OGR (via a string of errors along the way)

The basic pattern for OGR2OGR usage is given at http://www.gdal.org/ogr2ogr.html, with additional usage options for the SQL Server driver at http://www.gdal.org/ogr/drv_mssqlspatial.html. So, let’s start by just trying out a basic example to export the entire OGRExportTestTable from my local SQLExpress instance to a shapefile at c:\temp\sqlexport.shp, as follows: (change the connection string to match your server/credentials as appropriate)

ogr2ogr -f "ESRI Shapefile" "C:\temp\sqlexport.shp"
"MSSQL:server=localhost\sqlexpress;database=tempdb;tables=OGRExportTestTable;trusted_connection=yes;"

This will fail with a couple of errors, but the first one to address is as follows:

ERROR 1: Attempt to write non-point (LINESTRING) geometry to point shapefile.

SQL Server will allow you to mix different types of geometry (Points, LineStrings, Polygons) within a single column of geometry or geography data. An ESRI shapefile, in contrast, can only contain a single homogenous type of geometry. To correct this, rather than trying to dump the entire table by specifying the tables=OGRExportTable in the connection string, we’ll have to manually select rows of only a certain type of geometry at a time by specifying an explicit SQL statement. Let’s start off by concentrating on the points only. This can be done with the –sql option, as follows:

ogr2ogr -f "ESRI Shapefile" "C:\temp\sqlexport.shp"
"MSSQL:server=localhost\sqlexpress;database=tempdb;trusted_connection=yes;"
-sql "SELECT * FROM OGRExportTestTable WHERE shapegeom.STGeometryType() = 'POINT'"

This time, a new error occurs:

ERROR 1: C:\temp\sqlexport.shp is not a directory.

ESRI Shapefile driver failed to create C:\temp\sqlexport.shp

Although our first attempt to export the entire table failed, it still created an output file at C:\temp\sqlexport.shp. Seeing as we didn’t specify the behaviour for what to do when the output file already exists, when we run OGR2OGR for the second time it has now errored. To correct this, we’ll add the –overwrite flag.

ogr2ogr -f "ESRI Shapefile" "C:\temp\sqlexport.shp"
"MSSQL:server=localhost\sqlexpress;database=tempdb;trusted_connection=yes;"
-sql "SELECT * FROM OGRExportTestTable WHERE shapegeom.STGeometryType() = 'POINT'"
-overwrite

Running again and the error is now:

ERROR 6: Can’t create fields of type Binary on shapefile layers.

Ok, so remember that our original shapefile had two geometry columns. When you create a shapefile, one column is used to populate the shape information itself, while every other column becomes an attribute of that shape, stored in the associated .dbf file. Since the remaining geometry column is a binary value, this can’t be stored in the .dbf file. To resolve this, rather than using a SELECT *, we’ll explicitly specify each column to be included in the shapefile. You could, if you want, then omit the second geometry column completely from the list of selected fields. Instead, I’ll use the ToString() method to convert it to WKT, which can then be stored as an attribute value:

ogr2ogr -f "ESRI Shapefile" "C:\temp\sqlexport.shp"
"MSSQL:server=localhost\sqlexpress;database=tempdb;trusted_connection=yes;"
-sql "SELECT shapeid, shapename, shapegeom, bufferedshape.ToString(), bufferedshapearea  FROM OGRExportTestTable WHERE shapegeom.STGeometryType() = 'POINT'"
-overwrite

This should now run without errors, although you’ll still get a warning:

Warning 6: Normalized/laundered field name: ‘bufferedshapearea’ to bufferedsh

The names of any attribute fields associated with a shapefile can only be up to a maximum of 10 characters in length. In this case, OGR2OGR has manually truncated the name of the buferedshapearea column for us, but you might not like to use the garbled “bufferedsh” as an attribute name in your shapefile. A better approach would be to specify an alias for long column names in the –sql statement itself. In this case, perhaps just “area” will suffice:

ogr2ogr -f "ESRI Shapefile" "C:\temp\sqlexport.shp"
"MSSQL:server=localhost\sqlexpress;database=tempdb;trusted_connection=yes;"
-sql "SELECT shapeid, shapename, shapegeom, bufferedshape.ToString(), bufferedshapearea AS area  FROM OGRExportTestTable WHERE shapegeom.STGeometryType() = 'POINT'"
-overwrite

And now, for the first time, you should be both error and warning free:

So, now, just repeat the above but substituting the POINT, LINESTRING, and POLYGON geometry types, and creating three separate corresponding shapefiles:

ogr2ogr -f "ESRI Shapefile" "C:\temp\sqlexport_point.shp" "MSSQL:server=localhost\sqlexpress;database=tempdb;trusted_connection=yes;" -sql "SELECT shapeid, shapename, shapegeom, bufferedshape.ToString(), bufferedshapearea AS area  FROM OGRExportTestTable WHERE shapegeom.STGeometryType() = 'POINT'" -overwrite

ogr2ogr -f "ESRI Shapefile" "C:\temp\sqlexport_linestring.shp" "MSSQL:server=localhost\sqlexpress;database=tempdb;trusted_connection=yes;" -sql "SELECT shapeid, shapename, shapegeom, bufferedshape.ToString(), bufferedshapearea AS area  FROM OGRExportTestTable WHERE shapegeom.STGeometryType() = 'LINESTRING'" -overwrite

ogr2ogr -f "ESRI Shapefile" "C:\temp\sqlexport_polygon.shp" "MSSQL:server=localhost\sqlexpress;database=tempdb;trusted_connection=yes;" -sql "SELECT shapeid, shapename, shapegeom, bufferedshape.ToString(), bufferedshapearea AS area  FROM OGRExportTestTable WHERE shapegeom.STGeometryType() = 'POLYGON'" -overwrite

All done, right? Let’s just have a quick check on the created files to make sure they look ok. You can do this using ogrinfo with the –al option, which will give you a summary of the elements contained in any spatial data set. I mean, I’m sure they’re fine and everything, but…. hang on a minute:

ogrinfo -al c:\temp\sqlexport_point.shp

Here’s the POINT shapefile:

Initially looks ok, the shapefile contains 2 Point features – they’ve got the correct attribute fields, but both points seem to have been incorrectly placed at coordinates of POINT(0.0 2.0). Huh?

What about the LINESTRING shapefile:

This is even worse – even though we specified that only LineString geometries should be returned from the SQL query, the created shapefile thinks it contains 2 Point features (Geometry: Point, near the top). And those points both lie at POINT (0.0 0.0)…

And the POLYGON shapefile:

Same problem as the LineString – it’s effectively an empty Point shapefile.

The problem seems to be that, even though we’re only returning geometries of a certain type from the SQL query, we haven’t explicitly stated that to the shapefile creator, so OGR2OGR is creating three empty shapefiles first (each set up to receive the default geometry type of POINT), and then trying to populate them with unmatching shape types, leading to corrupt data. To explicitly state the geometry type of the shapefiles created, we need to supply the SHPT layer creation option for each shapefile as specified at http://www.gdal.org/ogr/drv_shapefile.html, by adding –lco “SHPT=POLYGON”, –lco “SHPT=ARC” (for LineStrings) etc. as follows:

ogr2ogr -f "ESRI Shapefile" "C:\temp\sqlexport_point.shp"
"MSSQL:server=localhost\sqlexpress;database=tempdb;trusted_connection=yes;"
-sql "SELECT shapeid, shapename, shapegeom, bufferedshape.ToString(), bufferedshapearea AS area  FROM OGRExportTestTable WHERE shapegeom.STGeometryType() = 'POINT'"
-overwrite
-lco "SHPT=POINT"

ogr2ogr -f "ESRI Shapefile" "C:\temp\sqlexport_linestring.shp"
"MSSQL:server=localhost\sqlexpress;database=tempdb;trusted_connection=yes;"
-sql "SELECT shapeid, shapename, shapegeom, bufferedshape.ToString(), bufferedshapearea AS area  FROM OGRExportTestTable WHERE shapegeom.STGeometryType() = 'LINESTRING'"
-overwrite
-lco "SHPT=ARC"

ogr2ogr -f "ESRI Shapefile" "C:\temp\sqlexport_polygon.shp" "MSSQL:server=localhost\sqlexpress;database=tempdb;trusted_connection=yes;"
-sql "SELECT shapeid, shapename, shapegeom, bufferedshape.ToString(), bufferedshapearea AS area  FROM OGRExportTestTable WHERE shapegeom.STGeometryType() = 'POLYGON'"
-overwrite
-lco "SHPT=POLYGON"

Unfortunately, that brings us round to almost exactly the same error as we first started with when creating the LineString and Polygon shapefiles (Gah! I thought we’d got rid of them!):

ERROR 1: Attempt to write non-linestring (POINT) geometry to ARC type shapefile

ERROR 1: Attempt to write non-polygon (POINT) geometry to POLYGON type shapefile

We are only selecting geometries of the matching type for each shapefile by filtering the query based on STGeometryType(), so why does OGR2OGR think that we are selecting other types of geometries? What’s more, we haven’t yet explained why the point shapefile (which was, after all, being populated only with point geometries), incorrectly placed both points at coordinates POINT(0.0 2.0). It seems that something is corrupting the results of the SQL statement.

And here’s the “Ta-dah!” moment. According to http://www.gdal.org/ogr/drv_mssqlspatial.html, when retrieving spatial data from SQL Server, “The default [GeometryFormat] value is ‘native’, in this case the native SqlGeometry and SqlGeography serialization format is used”. However, this doesn’t actually appear to hold true. SQL Server stores geometry and geography data in a format very similar to, but slightly different from Well-Known Binary (WKB). The SQL Server binary values for the two points in the OGRExportTestTable are:

0x00000000010C0000000000002A400000000000002440
0x00000000010C0000000000001C400000000000002840

The Well-Known Binary of these two points is , instead, as follows:

0x01010000000000000000002A400000000000002440
0x01010000000000000000001C400000000000002840

As you can see, they’re very similar – the coordinate values are serialised as 8-byte floating point binary values in both cases, but the MSSQL Server native serialisation has a different and slightly longer (i.e. one byte more) header. Thus, if OGR2OGR is expecting to receive one type of data, but actually gets the other, all the bytes will be displaced slightly. This could explain both why OGR believed it was receiving the incorrect geometry types and also why the point coordinates were wrong.

To correct this, rather than retrieving the shapegeom value directly, use the STAsBinary() method in the SQL statement to retrieve the WKB of the shapegeom column instead.

ogr2ogr -f "ESRI Shapefile" "C:\temp\sqlexport_point.shp"
"MSSQL:server=localhost\sqlexpress;database=tempdb;trusted_connection=yes;"
-sql "SELECT shapeid, shapename, shapegeom.STAsBinary(), bufferedshape.ToString(), bufferedshapearea AS area  FROM OGRExportTestTable WHERE shapegeom.STGeometryType() = 'POINT'"
-overwrite
-lco "SHPT=POINT"

ogr2ogr -f "ESRI Shapefile" "C:\temp\sqlexport_linestring.shp"
"MSSQL:server=localhost\sqlexpress;database=tempdb;trusted_connection=yes;"
-sql "SELECT shapeid, shapename, shapegeom.STAsBinary(), bufferedshape.ToString(), bufferedshapearea AS area  FROM OGRExportTestTable WHERE shapegeom.STGeometryType() = 'LINESTRING'"
-overwrite
-lco "SHPT=ARC"

ogr2ogr -f "ESRI Shapefile" "C:\temp\sqlexport_polygon.shp" "MSSQL:server=localhost\sqlexpress;database=tempdb;trusted_connection=yes;"
-sql "SELECT shapeid, shapename, shapegeom.STAsBinary(), bufferedshape.ToString(), bufferedshapearea AS area  FROM OGRExportTestTable WHERE shapegeom.STGeometryType() = 'POLYGON'"
-overwrite
-lco "SHPT=POLYGON"

Right, we are definitely making progress now. Trying ogrinfo again reveals that all of the layers have the correct number of features, or the appropriate type, and all have the right coordinate values. Yay!

There’s just one nagging thing and that’s the Layer SRS WKT: (unknown). When we converted from the SQL Server native serialisation to Well-Known Binary, we lost the metadata of the spatial reference identifier (SRID) associated with each geometry. This information contains the details of the datum, coordinate reference system, prime meridian etc. that make the coordinates of each geometry relate to an actual place on the earth’s surface. In the shapefile format, this information is contained in a .PRJ file that usually accompanies each .SHP file but, since we haven’t supplied this information to OGR2OGR, none of the created shapefiles currently have associated .PRJ files, so ogrinfo reports the spatial reference system as “unknown”.

To create a PRJ file, we need to append the –a_srs option, supplying the same EPSG id as was supplied when creating the original geometry instances in the table. (Of course, if you have instances of more than one SRID in the same SQL Server table, you’ll have to split these out into separate shapefiles, just as you have to split different geometry types into separate shapefiles)

The Final Product

So here’s the final script that will separate out points, linestrings, and polygons from a SQL Server table into separate shapefiles, via the WKB format but maintaining the SRID of the original instance, and retaining all other columns as attribute values in the .dbf file. Tested and confirmed to work with columns of either the geometry or geography datatype in SQL Server 2008/R2, or SQL Server Denali CTP3.

ogr2ogr -f "ESRI Shapefile" "C:\temp\sqlexport_point.shp"
"MSSQL:server=localhost\sqlexpress;database=tempdb;trusted_connection=yes;"
-sql "SELECT shapeid, shapename, shapegeom.STAsBinary(), bufferedshape.ToString(), bufferedshapearea AS area  FROM OGRExportTestTable WHERE shapegeom.STGeometryType() = 'POINT'"
-overwrite
-lco "SHPT=POINT"
-a_srs "EPSG:2199"

ogr2ogr -f "ESRI Shapefile" "C:\temp\sqlexport_linestring.shp"
"MSSQL:server=localhost\sqlexpress;database=tempdb;trusted_connection=yes;"
-sql "SELECT shapeid, shapename, shapegeom.STAsBinary(), bufferedshape.ToString(), bufferedshapearea AS area  FROM OGRExportTestTable WHERE shapegeom.STGeometryType() = 'LINESTRING'"
-overwrite
-lco "SHPT=ARC"
-a_srs "EPSG:2199"

ogr2ogr -f "ESRI Shapefile" "C:\temp\sqlexport_polygon.shp"
"MSSQL:server=localhost\sqlexpress;database=tempdb;trusted_connection=yes;"
-sql "SELECT shapeid, shapename, shapegeom.STAsBinary(), bufferedshape.ToString(), bufferedshapearea AS area  FROM OGRExportTestTable WHERE shapegeom.STGeometryType() = 'POLYGON'"
-overwrite
-lco "SHPT=POLYGON"
-a_srs "EPSG:2199"

And here’s the resulting shapefile layers loaded into qGIS:

I’m not going to go into detail about the (fairly radical) changes made in Windows 8, partly because many other people have already written about these, and partly because I’m very unqualified to do so – I’m finding things out myself as I go along. Instead, seeing as Windows 8 is all about the user interface and aimed at supporting touch-enabled devices, the main thing I was interested in was seeing what’s involved in developing a simple “Metro” application with a slippy map interface. This post is a summary of my initial investigations.

What’s Metro, anyway?

Metro is the (current) name of the new, touch-centric style that is exposed as the default interface in Windows 8. It’s certainly a radical departure from traditional Windows UI, and more resembles the interface you’d get on a mobile phone handset or other portable device. (This is not coincidence – the big growth market in the PC world at the moment lies not in desktops or laptops, but with tablet PCs, and it’s clear that Microsoft is designing Windows 8 to be run on devices that will “compete” against Apple’s iPad and Android tablets). Apart from the touch interface, there are other similarities as well – Metro applications are typically small in size, relatively simple, have a full-screen UI, are responsive (due to asynchronous method calls), and they can be packaged up and distributed via the “Windows Application Store” – a model much like Chrome’s web store, Android’s marketplace or Apple’s App Store.

So, I thought it would be nice to create a simple little Metro app that displayed a slippy map interface, possibly showed points of interest near to you (using the built-in geolocation), allowed you to plot routes etc. – fairly regular stuff. Since this is Microsoft Windows, after all, you’d think that this would be relatively easy to do with the Bing Maps API, right? Well….

Developing for Metro – Choose your Weapon

First things first – the core system component that powers Metro Apps is something called WinRT. WinRT contains APIs for handling devices, graphics, communications etc. that are exposed via COM interfaces. You can write Metro applications that reference these WinRT assemblies using a variety of languages – managed code (C#/VB) or unmanaged code (C/C++) with a XAML front-end, or using Javascript with an HTML/CSS front-end. The Windows 8 technology stack is summarised in this slide shown at the recent BUILD conference:

Bing Maps comes in lots of different flavours too, including (amongst others) a Silverlight control, a WPF control, and a Javascript control. Since WPF and Silverlight both make use of XAML for UI and managed code-behind, you might think that it would be relatively easy to port an application that uses the Bing Maps Silverlight/WPF control to Metro XAML and c# code-behind, but actually I’ve not heard of anybody successfully do this yet. Also, considering that the WPF Bing Maps control is still itself only in beta, and the Silverlight control has apparently been in stasis for several years, I certainly wasn’t about to commit any effort to making these work on Windows 8 when I’m not convinced about their future longevity….

Instead, I thought I’d opt for the HTML/Javascript approach – which are technologies I’m more familiar with anyway, and have a more mature associated Bing Maps AJAX control. However, even then, I’ve been surprised at the number of changes required to port a regular HTML/Javascript Bing Maps web application and get it to work in a Metro app.

HTML/Javascript in Metro – it’s all about Context

Most people think of HTML/Javascript as being exclusively used on the web – the two technologies being responsible for structure/client-side code of webpages respectively, and executed only within the context of a web browser. This, in itself, has historically made the Javascript execution environment somewhat sandboxed. Even though projects such as nodejs have shown that Javascript can also be used in other contexts, it has not been widely used for creating desktop applications before. (In a Windows context, Javascript is used for creating sidebar gadgets in Windows Vista/7,  but these have generally been little more than mini-webpages pinned to the desktop, and have little access to system resources)

In Metro, however, a Javascript application can be used to access core parts of the operating system (including the file system, networking, and other devices) by calling into the WinRT APIs. This sounds kind of dangerous, doesn’t it? Javascript, with its <script> injection, dynamic DOM manipulation et al. having access to your system resources? To minimise the risk of malicious code in this scenario, Windows 8 introduces different security contexts for Metro apps.

The main HTML landing page for a Metro app runs in the local context. It has access to the WinRT APIs, and can also do some other things not possible using Javascript running in a regular webbrowser, such as making cross-domain XmlHttpRequests. However, there are two main restrictions placed on pages running within the local context:

• You can only load locally-packaged scripts. i.e. you can’t include a <script> tag whose src attribute points to a file on a remote server.
• Any methods that attempt to change the DOM (i.e. by setting an element’s innerHTML or calling the document.write method) are sanitised.

Unfortunately, that means using the Bing Maps AJAX API is not possible from within a Metro app running within the local context, because the Bing Maps control needs to be referenced from a remote URL (in other words, you need to include the library using <script type=”text/javascript” src=”http://ecn.dev.virtualearth.net/mapcontrol/mapcontrol.ashx?v=7.0></script>), which is not allowed within the local context. Even if you created an offline copy of the Bing Maps AJAX library (which is not supported), it would still not work because of the way in which it manipulates the DOM of the HTML page.

So, what options do you have?

Option 1: Embed the Map Control in a iFrame using the Web Context

Although the main HTML page of a Metro app must run in the local context, you can have additional HTML pages that are loaded within the web context. When running in the web context, your HTML/JS code behaves almost exactly as it would do if loaded within a regular Internet Explorer browser – in other words, you lose the ability to tap into the WinRT APIs and other new Metro features, but you can include remote script references again as normal. For a full comparison of what’s possible in the local context and in the web context, refer to http://msdn.microsoft.com/en-us/library/windows/apps/hh465373%28v=VS.85%29.aspx

However, I wanted to embed a slippy map as the main frontend UI on my Metro App – I don’t want the user to have to navigate to a separate HTML page in the web context just to load the map. So, is there a way to get the best of both worlds – having elements loaded from within the web context displayed within the main page in the local context? One approach is to create a regular Bing Maps application in a separate HTML page in the web context, and then embed that in our main application in a separate iframe.

To do so, create an iframe in the main page in which the src element points to your Bing Map page, prefixed by  ms-wwa-web:/// to show that the content of the iframe is to be loaded in the web context. So, here’s a simple default.html page for a Metro App:

<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=1024, height=768" />
    <title>WinWebApp1</title>
    <!-- WinJS references -->
    <link rel="stylesheet" href="/winjs/css/ui-dark.css" />
    <script src="/winjs/js/base.js"></script>
    <script src="/winjs/js/wwaapp.js"></script>
    <script src="/winjs/js/ui.js"></script>
    <script src="/winjs/js/controls.js"></script>
    <!-- WinWebApp1 references -->
    <link rel="stylesheet" href="/css/default.css" />
    <script src="/js/default.js"></script>
</head>
<body>
    <header role="banner" aria-label="Header content">
        <div class="titleArea">
            <h1 class="pageTitle win-title" role="button" aria-label="Groups" tabindex="0">
                Metro Map App</h1>
        </div>
    </header>
    <div>
      <iframe id="mapIframe" src="ms-wwa-web:///map.html" width="1024px" height="400px"></iframe>
    </div>
</body>
</html>

and here’s the map.html page that is loaded within the web context of the iframe:

<!DOCTYPE html>
<html>
<head>
    <title>Map</title>
    <!-- We can include remote script because this page will be loaded in the web context (ms-wwa-web:///map.html)  -->
    <script src="http://ecn.dev.virtualearth.net/mapcontrol/mapcontrol.ashx?v=7.0" type="text/javascript"></script>
    <script type="text/javascript">
        var map = null;

        function initialize() {
            var mapOptions = {
                credentials: "AkeAFl99ZABCDEFG7Kb2D12345678lRQm8vnZMpfMV7HsfNqwertyuiopd",
                center: new Microsoft.Maps.Location(52, 0),
                mapTypeId: Microsoft.Maps.MapTypeId.road,
                zoom: 7,
                showLogo: false,
                showDashboard: false
            };
            map = new Microsoft.Maps.Map(document.getElementById("map"), mapOptions);
        }
        document.addEventListener("DOMContentLoaded", initialize, false);
    </script>
    <style type="text/css">
        body, html
        {
            margin:0;
        }
        
        #map
        {
          position:absolute;
          width:100%;
          height:100%;
        }
    </style>
</head>
<body>
    <div id="map"></div>
</body>
</html>

Hit F5 to debug the app and you’ll get something like this – a regular slippy map interface using Bing Maps, contained in an iframe that seamlessly integrates into the container of your Metro App.

So far so good. But the problem with this approach is that, although visually it may appear integrated, the content of the iframe is essentially isolated from the main application (that, after all, is the whole reason why the iframe is allowed to be loaded in the web context – if it could directly access the WinRT APIs available to the parent application you’d be exposing the security risks that the whole local context scenario is meant to avoid). So how do we go about creating interface elements in the parent container page that update (or are updated by) elements in the map iframe?

You can communicate between the host application running in the local context and the map iframe running in the web context by using the HTML5 postMessage method. If you search the internet, you’ll find plenty of examples of postMessage, but they’re very simplistic – most of the examples involve sending a single text string from one window to another and, when it’s received, simply alerting that message to the user. But when you actually want to try to expose entire interfaces between iframe windows, you’ll quickly create a mess of code – the postMessage method is, after all, designed for…. posting messages… so any data passing between iframes must be passed as text. You then need to setup a messageHandler on the receiving window that will act appropriately based on the text data received.

For example, I set up a simple row of button controls in my container HTML page to pan the map and zoom in/out.

<button onclick="Pan(-1,0);">Pan Left</button>
<button onclick="Pan(1,0);">Pan Right</button>
<button onclick="Pan(0,1);">Pan Up</button>
<button onclick="Pan(0,-1);">Pan Down</button>
<button onclick="ZoomOut();">Zoom Out</button>
<button onclick="ZoomIn();">Zoom In</button>

The methods attached to these buttons do not pan or zoom the map directly. Rather, they create an instruction that will be sent via postMessage to the map iframe. Fortunately, the postMessage API implemented in Internet Explorer 10 (which is the engine in which Metro Apps running in the web context are executed) accepts postMessages either as single strings or as JSON objects. So,  the “instruction” passed to the map is contained in a JSON object in which method is the action to perform on the map, and args contains any arguments to pass to that method. Notice that I specify that the target domain of the postMessage call lies in the ms-ww-web web context:

function Pan(dx, dy) {
    var xMsg = { method: 'pan', args: {x: dx, y: dy} };
    mapIframe.postMessage(xMsg, "ms-wwa-web://" + document.location.host);
}

function ZoomIn() {
    var xMsg = { method: 'zoomin' };
    mapIframe.postMessage(xMsg, "ms-wwa-web://" + document.location.host);
}

function ZoomOut(dx, dy) {
    var xMsg = { method: 'zoomout' };
    mapIframe.postMessage(xMsg, "ms-wwa-web://" + document.location.host);
}

Then, in the map iframe, I need to set up a listener to any postMessages received:

window.addEventListener("message", receiveMessage, false);

and, in the receiveMessage() callback, unravel and handle the requested JSON instruction (which is contained in the event.data parameter):

function receiveMessage(event) {
  switch (event.data.method) {
    case 'pan':
      var dx = event.data.args.x;
      var dy = event.data.args.y;
      var pos = map.tryPixelToLocation(new Microsoft.Maps.Point(map.getWidth()/2 + dx, map.getHeight()/2 + dy, Microsoft.Maps.PixelReference.viewport));
      map.setView({ center: pos });
      break;

    case 'zoomin':
      var currentZoom = map.getZoom();
      map.setView({ zoom: currentZoom + 1 });
      break;

    case 'zoomout':
      var currentZoom = map.getZoom();
      map.setView({ zoom: currentZoom - 1 });
      break;
  }
}

Here’s a diagram of what’s going on:

Bear in mind that all I’m doing is adding the simplest of interactivity here – just panning and zooming the map. The more functionality you want to add, the more complicated the data structures you have to pass in the postMessage call (consider things like the set of points returned in a route requested from a routing service).

Also, notice that at the moment I’m only doing one-way communication from the parent container to the map iframe. But there are plenty of situations in which you’ll also want to pass messages the other way – to notify the container app when a certain action has happened on the map for example. Thus you’ll also need to setup an event listener on the container page to listen to incoming messages from the map iframe. Using the nested iframe approach and postMessage calls to create two-way interfaces like this quickly becomes unmanageable…

Option 2: Use an AJAX Map Control that works within the Local Context

In order to be executed directly from an HTML page in the local context, a Javascript library must be loaded locally. So how about we ditch the Bing Maps control altogether and use an alternative control that can be run from a local script. How about Leaflet, say?

If you don’t know, Leaflet is a new open source AJAX map control from CloudMade – a company with very close associations to the Open Street Map project. In the past, developers looking for an open source slippy map control have tended to opt for OpenLayers – but Leaflet looks set to provide a more modern, lightweight alternative. In fact, even though it’s only a few months old, Leaflet already offers many features not found in commercial map controls from the likes of Bing and Google (including support for WMS services, <canvas> tiles, and extensible map objects). But, for the purposes of developing a Metro application, Leaflet’s best feature is that it’s a very small download, consisting of just a single .js file, one .css file, and a handful of images that you can include in your own project.

Fortunately, the Leaflet API also follows very similar syntax to Bing Maps / Google Maps and will seem familiar to anyone who’s used to the idea of pushpins, infoboxes, polylines and polygons. Within a few minutes, I was therefore able to rewrite my Bing Maps app to use Leaflet instead. And, because I was loading the Leaflet library from a local <script> resource I didn’t have to worry about embedding an iframe or any of that postMessage nonsense – I could incorporate the map directly in my main HTML page in the local context. Creating a basic Leaflet map involves the following:

var map = new L.Map('divMap');
map.setView(new L.LatLng(52,0), 7);

Then, with a few calls to the Bing Maps REST service I created a new L.TileLayer using Bing Maps road style tiles to match the look and feel of my previous attempt (by default, Leaflet uses Open Street Map tiles). The result looked like this:

Visually, not much difference, but the pushpin popup says it all.

Summary

I don’t know the performance difference between these approaches – I guess there must be an overhead in having to marshal postMessage calls between iframes but I don’t know if its significant or not. Anecdotally, using Leaflet in the local context seems to be more responsive than the Bing Maps iframe approach.

Better performing or not, the main thing to note is that it’s a lot easier to write manageable code using the second approach. And there’s the irony – until (if?) a native WinRT maps control comes out (or unless you can figure a better way to handle the local/web context dilemma than me), if you’re looking to develop a slippy map interface for a Windows Metro application it’s actually much easier to do so using a third-party map API such as Leaflet than it is to use Microsoft’s own Bing Maps API…