Customising CiviCRM Mailing Labels

So, today I had a break from spatial stuff to do some volunteer I.T. work for a local not-for-profit organisation. They use CiviCRM as their contact management system, and they wanted to modify the way that it produces mailing labels for mass mailings (i.e. real paper mailings, not the e-mailshot kind).

Adding a Custom Field to a Mailing Label

The first task was to modify the default fields displayed on the mailing label (name, address, postcode etc.) to include a custom field. This organisation delivers a lot of its local mail by hand rather than through the royal mail, so the custom field was used to assign addresses into a delivery walk, and this was to be printed on the label so it could be given to the appropriate deliverer.

Fortunately, CiviCRM uses a token replacement system, and every user-created custom field is assigned a unique token. The only problem in including the deliverer custom field on the mailing label  was finding out what the token corresponding to the appropriate field was.

I’m sure there must be a better way, but the quick solution that worked for me was to bring up the “Print PDF Letter for Contacts” action from a search result

On the next screen, click the “Insert Tokens” link at the top right to be presented with a modal dialog box listing all the names of all the available tokens in your system. Clicking on the name of a token will insert it into the PDF letter, allowing you to see the token behind it. In the screenshot below, for example, the Geo Code 1 field associated with each individual contact can be inserted into any mailing (or mailing label) using the {contact.geo_code_1} token.

Having identified the appropriate token for a custom field (in my case, it was {contact.custom_4} ), you can then add this to the mailing label template by going to Administer >> Configure >> Global Settings >> Address Settings and editing the Mailing Label Format to include the relevant token, as below:

{contact.addressee}
{contact.street_address}
{contact.supplemental_address_1}
{contact.supplemental_address_2}
{contact.city}{, }{contact.state_province}{ }{contact.postal_code}
{contact.country}
{contact.custom_4}

Sorting the Mailing List

Having ensured that the custom field was printed onto each label, the next step was to sort the list of labels based on the value of that field. As explained before, the mailings were to be hand-delivered, and it would save a lot of time sorting through the envelopes if they could be printed ready-sorted into batches for the appropriate deliverer, as specified by the custom field. A quick search of the internet reveals that sorted mailings are also required in other circumstances as well, such as sorting by ZIP code when submitting batch mailings to the U.S. Postal Service. However, although reported as a major bug in CiviCRM, the ability to sort mailing lists in anything other than alphabetical surname order of addressee is currently not possible. So I had to hack something together.

A snoop around revealed that the bulk of the work in preparing the mailing labels occurred in the script located at  \sites\all\modules\civicrm\CRM\Contact\Form\Task\Label.php. Within the CRM_Contact_Form_Task_Label class in this file, I created a new comparison function which would sort an array of the items to be included in the mailing list based on the custom_4 field, as follows:

function cmpCustomField($a, $b) {
  if ($a['custom_4'] == $b['custom_4']) {
    return 0;
  }
  return ($a['custom_4'] < $b['custom_4']) ? -1 : 1;
}

Through a bit of experimentation, I determined the best place to sort the records was after they had been run through the “Merge same address”/“Merge same household” scripts, but before being formatted for display. So I inserted a call to the PHP uasort function (sorts an associative array while keeping indexes) that used my comparison function at around line 330, as follows:

...
if ( isset( $fv['merge_same_household'] ) ) {
  $rows = $this->mergeSameHousehold( $rows );
  $individualFormat = true;
}

// INSERT CUSTOM SORT HERE
uasort ( $rows , array($this, 'cmpCustomField'));
    
// format the addresses according to CIVICRM_ADDRESS_FORMAT (CRM-1327)
require_once 'CRM/Utils/Address.php';
foreach ($rows as $id => $row) {
...

The mailing list labels were now produced in nicely ascending order based on the custom field.

Changing the Mailing Label Paper Template

CiviCRM supports a handful of common avery label formats out-of-the-box, but this organisation had recently purchased a bulk load of non-standard labels (A4, 24 labels per page, 8x3) which did not fit any of the supplied templates. They therefore couldn’t be used.

To create a new paper template, changes to two files must be made (both called “Label.php”, but in different directories).

Firstly, in the \sites\all\modules\civicrm\CRM\Utils\PDF\Label.php look for the $averyLabels array and add a new element to the array with the margins and other dimensions of your label sheet. Here’s the settings I used for my new paper (NX and NY are the number of columns and rows of labels on the sheet, width and height determined by dividing the overall dimensions of the page by the number of labels, and the margins determined mostly by guesswork):

$averyLabels = array (
  ...
  'Custom_3x8' => array(
    'name' => 'NGP_3x8',
    'paper-size' => 'A4',
    'metric' => 'mm',
    'lMargin' => 5,
    'tMargin' => 10,
    'NX' => 3,
    'NY' => 8,
    'SpaceX' => 0,
    'SpaceY' => 0,
    'width' => 70,
    'height' => 37.125,
    'font-size' => 9
  )
);

Having defined the settings for the Custom paper template, to actually get these settings to be selectable when producing mailing labels, you also need to edit the file at \sites\all\modules\civicrm\CRM\Contact\Form\Task\Label.php

Near the top is a buildQuickForm() function that contains an array of templates to populate a select list. Add the name of your new paper template to the end of the array, exactly as you defined it in the $averyLabels array above:

$label = array(
  "5160" => "5160",
  "5161" => "5161",
  "5162" => "5162", 
  ...,
  "Custom_3x8" => "Custom_3x8"
);

Save the file, and your new template will magically become available for use:

A few tweaks…

Finally, there were just a few tweaks to be made. The new paper template should be always be selected by default for mailing labels (they’d bought reams of the stuff, so it was going to be used for all labels for quite a while yet), and also the “merge contacts with the same address” should always be selected. Adding these defaults was a simple matter of amending the $defaults array declared in the setDefaultValues() function near the top of the \sites\all\modules\civicrm\CRM\Contact\Form\Task\Label.php file.

function setDefaultValues()
{
  $defaults = array();
  $defaults['do_not_mail'] = 1;
        
  // Modified by AA
  $defaults['merge_same_address'] = 1;
  $defaults['label_id'] = "Custom_3x8";
        
  return $defaults;
}

I’m no CiviCRM expert, and I’ve certainly no idea if the approach I took to solving any of these problems represents “best practice” in any way, but it seems to have got the job done and met the immediate needs of my client, so hopefully it might be helpful to somebody else too!

PHP and UTF-8 BOM (Or, why do my webpages start with  )

Like many developers, I write code for a variety of platforms, using a variety of platforms. I write C# on an iBook, PHP on a Linux VM running under Windows, Javascript on my desktop, and HTML blogposts on my mobile phone. I’ve been known to write C++ on my microwave. And I write SQL snippets on the back of my hand.

Unsurprisingly, this causes a few headaches sometimes, both from configuration differences between various development environments and also the limited amount of memory in my brain to remember the nuances of all these languages. To quote A.A.Milne’s Winnie the Pooh: “I am a bear of very little brain…”.

I recently wrote some PHP for the first time in ages, and noticed some of my pages were appearing on one development machine, in some browsers, preceded by the characters . These characters didn’t show up when editing the pages, and they didn’t show up at all when served from a different server or when viewed in some other browsers.

Initially, I thought that it was something to do with not having configured the correct character set in the response header (which is generally the main cause of garbled characters appearing in webpages), but, checking the response header it seemed ok – I was outputting UTF-8 as desired:

header('Content-type: text/html; charset=UTF-8') ;

And browsers viewing the page were correctly auto-detecting the character encoding as UTF-8:

Then I checked the configuration of the server, which was also set up with Unicode support correctly. And then I checked the encoding of the PHP scripts themselves, which were all encoded using Unicode UTF-8 – (Windows Codepage 65001). So far, everything seemed consistent, so where were those garbled characters coming from?

UTF-8 with or without signature – your choice. (Or not).

The reason, as I found out, was that one of my development environments (Visual Studio – from which I’d made the most recent edits to the affected pages) was configured to save UTF-8 encoded files with signature. Here’s the options for Unicode character encoding in Visual Studio, showing UTF-8 both with and without signature (notice that they’re both the same codepage – 65001):

There seems to be very little convention or standardisation as to the use of this “signature”. I hadn’t really come across this problem before because I generally use Eclipse for PHP development. The encoding options there are shown below:

Notice that, although there are several flavours of UTF-16 available in Eclipse, there is only version of UTF-8, which is equivalent to Visual Studio’s without signature.

Then again, here are the options in Windows Notepad (yes, I use that sometimes as well). As in Eclipse, there is only one choice of UTF-8, but this time the sole option available  provides the opposite behaviour – always saving UTF-8 with signature:

BOM BOM BOOM!

The optional “signature” in question is the Byte-Order Marker, or BOM. A byte-order marker is required for multibyte encoded data, including UTF-16, to indicate big-endianness or little-endianness – the order in which bytes are arranged. All of the save dialogs above give you the choice for specifying the byte order for Unicode UTF-16, since in a multibyte format the byte order matters. However, for UTF-8, which uses only a single byte for each character (that’s what the “8” stands for – 8 bits = 1 byte) a BOM is not required and doesn’t really make sense.

Even though UTF-8 always uses the same byte-order, a UTF-8 encoded file can begin with the bytes EF BB BF, which merely signifies that it is in UTF-8 format. It’s not really a BOM, hence why Visual Studio calls it a “signature”. The problem is that some clients don’t expect UTF-8 to have a BOM and, as it turns out, the PHP engine is one of them. At least, some builds of the PHP engine. One of my PHP servers, running on a linux machine, interpreted the UTF-8 file with signature fine, whereas another, running under Windows, tried to display the leading bytes as content on the page, which is how you end up with .

The combination of different default encoding behaviours across different editors combined with different server/browser behaviours when interpreting UTF-8 files with BOM means that this problem can be a little tricky to diagnose.

This is reported as a PHP bug at http://bugs.php.net/bug.php?id=22108, but the workarounds are actually quite straightforward (once you know what the problem is!):

• If you’re using Visual Studio, make sure you save your PHP files as UTF-8 without signature. If you’re using Eclipse, this is the default anyway.
• Compile your PHP with the –enable-zend-multibyte option, which will correctly parse the BOM at the start of the file
• If you don’t need unicode at all, you could use ISO-8566-1, or another non-UTF-8  encoding

The 4th Dimension – Creating Dynamic Animated Tile Layers in Bing Maps (AJAX v7) – Part 1

Many people (whether influenced more by Einstein’s special theory of relativity or the TARDIS) think of space and time as intricately linked dimensions; we describe the position of an object in space using three dimensions, and time represents the fourth dimension. Doctor Who, in his multi-dimensional policebox, is capable both of travelling throughout the universe and throughout the ages.

The concept of four-dimensional spacetime was challenged recently by a group of scientists who argue that time, in itself, is not a fundamental entity – it is merely a measure of the numerical order of changes in space. Nevertheless, whether dimension or not, time is a crucial factor to consider when interpreting spatial data. We not only need to know where something is, but when it was there.

Time and Maps

One common question asked on the MSDN Bing Maps forum concerns the timeliness of the base map data (particularly that of aerial photographic imagery) – “When was this image taken?” and “When will it next be refreshed?”. “Why isn’t this new building being shown or this recent housing development included?” etc. In many cases, as in the motivation behind these types of questions, we expect a map to display a single, most up-to-date current view of an area. However, this isn’t always the case. On occasions, we want to be able to navigate and explore a map both spatially and temporally – north, south, east, west, yesterday, and tomorrow; back into history and forwards into the future.

In fact, we all see and interpret this sort of time-dependent spatial data on a daily basis in the form of the weather forecast. I want to know what the weather will be like in Norwich at 3:00pm tomorrow:

In a static display medium, such as a newspaper, maps representing changing data over a period in time are generally shown as a series in a panel-display like that above. But what would be the best way to display changing time-based data on an interactive medium, such as Microsoft Bing Maps or Google Maps?

The most obvious method would be to have some sort of dynamic tilelayer – the background image of the map– that could be refreshed and updated, either on a scheduled interval or via some user control. Before writing this article I had a quick look (using a popular internet search provider^TM) to see if any library already existed to provide this functionality. There are a number of javascript libraries around that deal with timeline-based javascript animation, including Mashi and $fx. However, these tend to create animated effects by gradually-changing CSS styles affecting the position or opacity of existing elements. But to have a dynamic Bing Maps tilelayer we don’t want to change the style of the image elements on the page – we need to change the underlying images themselves.

So, having been unable to find any existing libraries or other resources on the topic, I started experimenting myself. In this first post of a series on the subject of creating dynamic maps, I’ll look at the most basic implementation of an animated tilelayer, which is…

Animated GIFs (or, flashback to the Internet circa. 1995)

Let’s start with the basics: to create a new tile layer in Bing Maps, you call the TileLayer constructor method specifying four properties of the tile layer in the supplied TileLayerOptions parameter: tile source, opacity, visible, and zIndex. Like this:

var tileOptions = {
  mercator: new Microsoft.Maps.TileSource({ uriConstructor: http://example.com/{quadkey}.png }),
  opacity: 0.3,
  visible: true,
  zIndex: 25
};

var tilelayer = new Microsoft.Maps.TileLayer(tileOptions);

The {quadkey} placeholder specified in the tilesource uriConstructor is dynamically replaced with the quadkey of the requested tile, which should then form a URI string to a valid image tile to be placed in the appropriate position map tile layer. So what if we constructed a set of animated GIFs showing the changing weather pattern for each tile, numbered by quadkey, and pointed the tilesource to these tiles? That way we could create a very simple tile layer that automatically animated through a series of time-based frames. The GIF animations themselves could be dynamically-created on the server-side based on data behind-the-scenes as necessary.

Animated GIFs got a bad reputation through their massive overuse on the internet to create irritating animations such as this:

Certainly, it’s been a very long time since I’ve tried to use GIFs in almost any context: the many negatives associated with the GIF file format include that it is a proprietary format which generally leads to large file sizes but with only limited 8-bit colour depth, and only basic support for transparency. In almost all circumstances, PNG or JPG is a better choice of images on the internet. However, among all the commonly-used image formats – JPG, PNG, TIF, BMP, etc. – GIF files are still the only image format that supports multiple frames of animation and are widely supported across all browsers. So maybe we shouldn’t rule them out completely…

Creating an Animated Weather Map

Having established weather forecast maps as a recognisable and very practical example of the presentation of time-based spatial data (and also one for which sample data was readily available from the NOAA), I decided to use this as the example on which to base my application. To begin with, I wrote a PHP script that connected to the NOAA radar web service and downloaded radar imagery for the whole of the contiguous U.S., like this:

To put this image in context, here it is again overlaid directly onto a Bing Maps base map:

I set up a cron job to call this PHP script every 15 minutes, and stored the results returned from the web service to build up a repository of images across a period of time. Then I cut each image up into a set of 256px x 256px tiles that matched the Bing Maps quadkey tiling system. After three hours passed, I had a set of 12 timestamped folders, each of which contained 385 tiles (I created tiles at zoom levels 5, 6, and 7 across the mainland U.S.), much like this:

Then, using the GIFEncoder class which can be found as part of the PHP Video Toolkit project, I looped through each folder and created a set of animated GIFs displaying the radar measurements for each tile over the last three hours. The following tile, for example, shows tile 021323 for the period 13:00 – 16:00 on 17th May 2011:

Then, by creating a tilelayer with a tilesource pointing to the location of my repository of animated GIF images, I was able to get a first shot at an animated weather map. Sadly, WordPress won’t let me insert an iframe or javascript here to show the finished product, but here’s a short video clip that demonstrates me navigating around the map as it refreshes:

Bing Maps Animated Weather Map

And, you can try it online, here: http://www.a3uk.com/demos/animatedgiftilelayer/

The advantages of using animated GIFs are:

• Very easy to set up
• Supported across most browsers, with little additional processing required by the client
• Once each tile is fully downloaded to the client, it can be cached
• GIF files can be created dynamically on the server

However, there are significant limitations with this method:

• Each individual GIF tile may be of considerable size – the single 256px x 256px tile above containing 18 frames of animation is 109kB. No frame can be displayed until the entire image for that tile has finished downloading.
• The animation cannot be controlled by the client at all – the duration between frames, the looping behaviour etc. is all pre-baked into the GIF file, and there is no way to stop, rewind, restart, or speed up the framerate, for example.
• There is no way (other than perhaps displaying a tiny timecode in the corner of every tile) to know what time period is represented by the frame currently being viewed.
• Each tile is completely independent, and is downloaded and animated separately from the others. The animation for each tile begins as soon as that particular tile image is loaded in the browser, so the animation of tiles in any given map view can become out-of-sync with each other.

In the next post, I’ll look at some alternative methods that overcome some of these limitations and allow you to programmatically control animated Bing Maps tilelayers using javascript.