Wednesday, November 19, 2014

How to monitor HTTP headers in near real-time

Unix has many tools that you can stitch together to do useful things.

For example, if you want to watch your site headers in near real-time you can use watch and curl.

You might want to do that if you need to build information headers (like X-Cache-Info) and you want to quickly try out different ideas.

In my case I use it during a patch window to monitor when our site's proxy server has changed so that the (now inactive) proxy can have updates applied.

The command that you run in a terminal is:

watch "curl -I -s"

-I is headers only, while -s is silent. It'll print the headers every two seconds by default. If you want to update every second it is:

watch -n 1 "curl -I -s"

Thursday, October 23, 2014

How to ignore alihack requests using your nginx config

If you are getting the following error in your Rails app:

   unexpected token at 'alihack<%eval request("")%> can be blocked in your nginx config.

The following snippet can be placed inside the server block and returns a  '400 Bad Request' with a text message in the body.

A big assumption here is that your app is not using json - it blocks all JSON PUT requests. This could be refined to check for third header that your own app sets, if that is a problem.

You could change this to a redirect or a json response if you want (commented out below).

# ali.txt attempt on any URL
if ($content_type = "application/json") { 
  set $ali_txt JSON; 

if ($request_method = PUT) {
  set $ali_txt "${ali_txt}_PUT"; 

if ($ali_txt = JSON_PUT) {
 return 400 "Bad request - ignoring"; 
 # return 444;
 # return 301

This should not be placed inside a location block.

Sunday, September 21, 2014

Radio NZ Browser and OS stats for September 2014

This is the September 2014 release of browser and OS stats for and The Wireless.

Overall, the Chrome browser is increasing its market share, and the shift from desktop to mobile continues. Windows has dropped to under 50%.


Browser 09/2014 03/2014 11/201306/201211/201111/201011/200911/2008
Chrome 30.8 25.23 23.7 14.6 13.8 8.75 4.2 1.47
IE 18.9 24.02 26.337.541.250.65663
Safari 18.8 18.94 17.5217.35.613.1105.66
Firefox 14.5 14.81 16.619.823.225.5227.527.73
Android 9.0 11.71 117.5
Opera 0.4 0.46 0.690.

IE is still in decline, and IE6 is 0.22%, IE7 1.2%, IE8 14%, IE9 19.2%, IE10 12.9%. IE11 is at 51%.

At The Wireless things are very different: Chrome 45%, Safari (in app) 15%, Firefox 13%, Safari 12.6%, IE 7.4%, Android 4.7%

Operating System

OS 09/2014 03/2014 11/201306/201211/201111/201011/200911/2008
Windows 48.9 52.9 5867.3728184.889.3
iOS 18.4 16.6 147.88
Android 17 16.5 13.77.793.60.30.020
Mac 11.7 10.7 13.414.715.614.212.68.5
iPad 2.40.6300
Linux 1.6 1.9
Win Phone 0.57 0.44 0.39
iPod 0.50.350.220.08

And at The Wireless, Windows 40.9%, iOS 22.8%, Mac 18.7%, Android 15%, Linux 1.4%, Win Phone 0.6%


In June 2012 mobile was 16% of our traffic. In November 2013 it was 29%. In March 2014 it was 23.4% mobile and 10.8% for tablet, making 34.2%. Now it is 61.8%, 26.3% mobile and 11.8% for tablet.

The most popular devices are still the same as last time - iPad (24.8%), iPhone (22.7%), and Samsung GT-I9300 Galaxy S III (3.9%)

Over at The Wireless 39% of the traffic is mobile, with popular devices being iPhone (41%), iPad (16.9%) Galaxy S III (4.3%).

Wednesday, September 17, 2014

How news events impact web traffic

This short post shows how news events impact website traffic on, and how we handle it technically.

On 18 August Guyon Espiner interviewed the Prime Minister about the claims in the book Dirty Politics. Within 10 minutes of the interview airing and being posted on the site traffic jumped to nearly twice normal levels, and stayed up for 12 hours. The interview was the most listened to audio on our site with around 18,000 listens.

On 26 August we saw a huge peak as our story about Manchester United signing Di Maria was linked from a UK news aggregator. This sometimes happens with our international coverage because our news cycle is 12 hours different from the UK.

Judith Collins resigned on 30 August and while there was a slight increase in traffic on the day, the interviews the following week were a big driver of visits.

On 15 September Morning Report interview the PM about the GCSB, which was followed by Glen Greenwald. He was interviewed again by Kathryn Ryan. These three interviews were the most listened to that day, and the next.

Against these peak events we've seen an overall increase in baseline visits to the site. This is partly to our excellent election section where the news team has put a lot of work into the coverage, and also to our election interview collection. The biggest stories have been our running Fact Or Fiction feature and the Poll of Polls where Colin James analyses the most recent Polls. The leader video profiles have also been popular.

There was a big peak on Tuesday 16 September as people came to view the live stream of examination of the Colossal Squid at Te Papa.

And then there are the regular visits to audio and other news coverage right across the whole site.

The Technical Side

Some of these fast-moving events can cause sudden and large increases in traffic, and that presents some technical challenges.

Firstly, the news team have to be able to update the story quickly and often while the site is under high load. We have a publishing system that allows our newsroom editing system to update our CMS directly with the press of one key. No messing about with HTML and image placement and a single story can be updated in under 5 seconds. The news subs can also update up to 100 stories (if they needed to) in under 30 seconds.

As an aside, we can typically publish audio interviews within 5 minutes of broadcast.

The traffic peaks are handled using a front-end cache, although this is setup to still allow frequent updates. We use Varnish which is reputed to be able to easily handle 10,000 requests a second, although we've never gotten to that number.

The content management system - our own bespoke design, built with the Ruby on Rails framework - has been optimised to have quick page load times, even over dial up and slow mobile connections.

All of these makes it a breeze (IMHO) to deliver news and audio content quickly to an increasingly demanding audience.

Saturday, August 2, 2014

Do you have the right CMS?

I have evaluated dozens of content management systems over the last 20 years, as well have having built several myself. The answer to the question, do we have the right CMS, is almost always going to be no.

It is no because your business will always be changing, and there will be a gap between what you want to do, and what your CMS will let you do.

Given this gap, the question that inevitably follows is, would be better off with something else. That depends on how big the gap is, and whether it can be closed (and the cost).

There are four things a CMS needs to do.

1. Product Delivery

The most obvious is that it must allow you to deliver your product. This might be a blog (writing), or selling things (commerce), or a large media site like the one I manage (news, features and audio story telling).

2. Business Processes Support

The second is that it must support a streamlined business process. Note that I am not saying it should support your current business process. It might already do that, and that might be a problem - it might be limiting your ability to improve and streamline the way you do business. 

It might even be, and I have seen this many times, that the functionality of the CMS forces you to work in a way that is inefficient or convoluted. Technology should not dictate to you how to run your business.

3. Systems Integration

It must also integrate with any of your existing business systems - and these are probably systems that you cannot easily replace. An example from my own company is how we've integrated our newsroom system - used by over 100 people daily - with the CMS.

4. Innovation

The last, and probably the most important is, does it allow you to continually innovate in the first three areas. Can you launch new products? Can you improve your business processes? Can you improve the way existing systems interact with the CMS?

If your CMS allows you to do these things, then you are probably in good shape but for one exception: cost. What is the cost of being able to achieve the above four points?

There are three broad classes of CMS, and each has it own cost structure.

1. Proprietary

The CMS is a 'black box'. You use it, but you cannot see or modify the source code. Any changes have to be done by you the vendor who 'owns' the code, at a cost and timeframe they determine, hopefully in consultation with your. The term 'charge like a wounded bull' comes to mind.

2. Supported Open Source

In this case you can view the code, but you must still use your contracted vendor to make changes and add new features. Some prefer this model because they have the best of both worlds - access to the code, and some performance guarantees (based on losing the ability to change the code themselves).

3. Fully Open Source

A fully open system means you, or anyone you choose, can modify your CMS code, although this can be at the expense of reliability and performance if the change processes are not carefully managed. You may choose to work with one vendor.

The first two models can be problematic because the vendor's system will evolve based on their customers' needs, and this may not line up with your own individual needs. In RNZ's case we changed from a supported open source solution that we'd used successfully for 5 years because the system's performance and content management focus was moving away from media organisations.

We now follow the third model for the CMS that drives In the last 30 days we have made over 100 changes to the CMS - some visible to the public, most to improve the administration section our staff use. We have also reduced the application's response time by 30% and a our page speed (according to Google) by 25% (see dip in the graph below).

In my experience that pace of change is next to impossible (at any reasonable cost) with the other two models. (We also have periods of little change, consolidation and reflection.)

Should you change your CMS?

You'll need to find out how much will it cost you to migrate to the new system, and determine if this will be paid back to you by the improvements you'll get from the change. Those improvements should include the four factors I outlined above.

In making a decision it can be useful to look at what your competitors are doing. In my field - media - the companies that are disrupting and innovating in this space all use custom CMSs built with open source software. They control the platform and in many cases the infrastructure too. I think that says a lot.

Which ever model you choose, you need to balance risk, innovation, internal capabilities (e.g. programming and infrastructure skills) and cost. I am happy to answer questions in the comments.

Monday, June 2, 2014

8 years of data

I have just published a Google doc of eight years of Operating System, Browser and Mobile data for

I have also included a couple of graphs such as this one showing the decline of desktop use (click to see a larger version).

A graph showing the decline of desktop use (65% in 2006) compared with mobile (35%)
There are other interesting trends - the decline of IE, the decline of the Windows desktop OS, and the rise of Chrome (all as a percentage of the market) are also evident in the data.

Graph showing the decline of IE, and the rise of other browsers such as Chrome and Firefox

A graph showing the decline in Windows from nearly 100% to just of 50%. iOS, android and OS takng much of that.

I have released this data under a Creative Commons Attribution-ShareAlike 4.0 International License, so please share links to anything you do with it in the comments below.

Sunday, May 18, 2014

A visual aid to show you which media query is active

I've found it handy during the development of responsive CSS to know what stylesheet or media query is active.

The following is an example of how to dynamically add labels to a page.

@media screen and (max-width:600px) {
  .site-time p::after { 
    content:" (0-600px)";

@media (min-width: 601px) and (max-width: 800px) {
  .site-time p::after { 
    content:" (601-800px)";

I've picked an element that was in every variant of the site layout, and appended some text indicating the active sheet.

Sunday, March 23, 2014

Radio NZ Browser Stats - March 2014

Here is another lot of browser and OS stats for and The Wireless.

Size of Content Library

The audio library now contains 27,000 hours of material, all of it searchable from That is 179,000 items, most of which are downloadable and embeddable.

We also just partnered with DigitalNZ who now have a copy of most of our audio metadata, and update it each night with new content.


Browser 03/2014 11/201306/201211/201111/201011/200911/2008
Chrome 25.23 23.7 14.6 13.8 8.75 4.2 1.47
IE 24.02 26.337.541.250.65663
Safari 18.94 17.5217.35.613.1105.66
Firefox 14.81 16.619.823.225.5227.527.73
Android 11.71 117.5
Opera 0.46 0.690.

IE is still in decline, and IE6 is 0.32%, IE7 1.86%, IE8 25%, IE9 20.7%, IE10 13%. IE11 is new at 37%.

At The Wireless things are very different: Chrome 37%, Safari (in app) 20%, Firefox 13%, Safari 12.5%, IE 11.3%, Android 5.0%

Operating System

OS 03/2014 11/201306/201211/201111/201011/200911/2008
Windows 52.9 5867.3728184.889.3
iOS 16.6 147.88
Android 16.5 13.77.793.60.30.020
Mac 10.7 13.414.715.614.212.68.5
iPad 2.40.6300
Linux 1.9
Win Phone 0.44 0.39
iPod 0.50.350.220.08

And at The Wireless, Windows 42%, iOS 26.5%, Mac 18.3%, Android 11%, Linux .78%, Win Phone 0.48%


In June 2012 mobile was 16% of our traffic. In November 2013 it was 29%. It is now 23.4% mobile and 10.8% for tablet, making 34.2%.

The most popular devices are the same as last time - iPad (25%), iPhone (22%), and Samsung GT-I9300 Galaxy S III (5%)

Over at The Wireless 38.3% of the traffic is mobile, with popular devices being iPhone (51%), iPad (17%) Galaxy S III (5%).