November 2013 Browser Stats for Radio NZ

Here it is once again - my roundup of browser and OS stats for www.radionz.co.nz. This time I'm including stats from The Wireless.

Size of Content Library

The audio library now contains 25,000 hours of material, all of it searchable from www.radionz.co.nz/audio. That is 167,000 items, most of which are downloadable and embeddable.

Browsers

Browser	11/2013	06/2012	11/2011	11/2010	11/2009	11/2008
IE	26.3	37.5	41.2	50.6	56	63
Chrome	23.7	14.6	13.8	8.75	4.2	1.47
Safari	17.52	17.3	5.6	13.1	10	5.66
Firefox	16.6	19.8	23.2	25.52	27.5	27.73
Android	11	7.5
Opera		0.69	0.7	0.9	1.02	1.08

IE is in decline, and IE6 is 0.6%, IE7 3%, IE8 31%, IE9 22.6%, IE10 42%.

At The Wireless things are very different: Chrome 35%, Safari 16%, Safari (in app) 16%, Firefox 15%, IE 9.4%, Android 5.5%

Operating System

OS	11/2013	06/2012	11/2011	11/2010	11/2009	11/2008
Windows	58	67.3	72	81	84.8	89.3
iOS	14	7.88
Android	13.7	7.79	3.6	0.3	0.02	0
Mac	13.4	14.7	15.6	14.2	12.6	8.5
iPhone			2.5	1.4	0.56	0.19
iPad			2.4	0.63	0	0
Linux	1.8	1.27	1.53	1.4	1.45	1.72
Win Phone	0.39
iPod			0.5	0.35	0.22	0.08

And at The Wireless, Windows 40%, iOS 26%, Mac 20%, Android 10%, Linux 1.9%, Win Phone 0.55%

Mobile

When I last published stats in June 2012 mobile was 16% of our traffic. It is now 29%.

The most popular devices are iPad (27%), iPhone (21%), and Samsung GT-I9300 Galaxy S III (6%)

Over at The Wireless 36% of the traffic is mobile, with popular devices being iPhone (47%), iPad (21%) Galaxy S III (4%).

Social

20% of referrals to radionz.co.nz come from social sites (and 35% from Google news).

73% of referrals to thewireless.co.nz come from social sites.

Why you should allow time (and pay) for refactoring of code

At Radio NZ we engage outside contractors to help us build the web applications we use. The most public of these apps is our website, and internally there is a large database system that is used company-wide.

One of the challenges of writing code is that over time the codebase tends to become rundown. This can be made worse if many people are working on that code. Differences in coding styles, changes in best-practices over time, current (and past) fashion; these all contribute to software rot.

That is not to say that the code was bad when it was first written, or that the original developer did not know what they were doing. Most likely, they did know what they were doing at the time, based on what they knew at that time. Perhaps there were other contraints (such as limited time). Someone did there best, at the time.

Refactoring is a critically important part of maintaining a software application.

Wikipedia describes refactoring:

Code refactoring is a "disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior", undertaken in order to improve some of the nonfunctional attributes of the software. Advantages include improved code readability and reduced complexity to improve the maintainability of the source code, as well as a more expressive internal architecture or object model to improve extensibility.

It goes on to quote Joshua Kerievsky:

By continuously improving the design of code, we make it easier and easier to work with. This is in sharp contrast to what typically happens: little refactoring and a great deal of attention paid to expediently adding new features. If you get into the hygienic habit of refactoring continuously, you'll find that it is easier to extend and maintain code. — Joshua Kerievsky, Refactoring to Patterns

Some will say, "if it ain't broke don't fix it".

Apply that to your car or your house and gardens and see where it gets you. Where it gets you is: broken down in the middle of nowhere, and your phone battery is flat because who needs to charge those things anyway. Or a house that has to be re-clad because you did not repaint it early enough, and the garden, well, I think we have all seen those.

Your code is a valuable asset. It contains business logic - knowledge about how things should work that have been built up over time, and codified in the software. You paid for the process that converted that knowledge to code.

The source code repository also contains (or should contain) business intelligence about why certain decisions were made, and the evolution of the code over time. This is vital information for future developers. If you do not have a code repository, please look for another field of work. Now.

Refactoring has a cost attached. It takes time. But at its core, refactoring is about reducing future costs.

You must accept the cost of refactoring, and make time for your developers to do this work.

The benefits are twofold:

1. Better quality code that costs less to maintain over time.

2. Your staff will be more productive.

On point two, everyone wants to have pride in their work. For programmers, refactoring code is a key part of the process of remaining connected to the code, and experiencing joy in their work. Many studies have shown that happy staff are more productive, on top of the gains of working with quality code.

At Radio NZ I encourage the developers who work for us to suggest refactorings, and I allocate the time for them to do the work because I see this as an investment in the future. The person who has to maintain this code might be me. Or it might be you. 

Forcing people to continue to work on code they know could be improved without letting them make those improvements is bad management.

Some of the code in your system will be hard to refactor. It is probable that this code was put together quickly, possibly without tests. Don't let this put you off. Read and understand the concept of technical debt.

Also, watch this excellent presentation where Katrina Owen refactors some code that has no tests. Watch it a second time if you don't quite get it. If you are technical, read Martin Folwer's book.

If you use Ruby and/or Rails read the book Practical Object-Oriented Design in Ruby. The approach outlined in book has fuelled many refactorings in our projects.

On point about tests. Having good tests is necessary to facilitate refactoring. Without tests you have no way to know for sure that the refactored code behaves the same as the original code.

Chances are, if you don't already have good tests, you will already being seeing application errors after changes to the code, and that these will be hard to find and fix. You'll also see new bugs appear as the direct result of fixing an existing bug. Yes, just like Novapay.

I suggest you do three things:

1. Right now, go and ask your key developers, is there some part of the application that is wasting their time and should be refactored. Get an estimate of how much time is required, and schedule it.

2. Start a discussion with your developers about the overal design of the app. Does it support where the business is now? Is it making future enhancements difficult, or reducing productivity and/or job satisfaction?

3. And more broadly, find out from from each of your developers three things that they feel make it hard for them to produce quality work. Don't put your own definition on quality, just ask the question. Make a commitment to fix these things.

The New Radio New Zealand Website

After three years of work, the new Radio New Zealand website is about to go now live.

As well as a new design, we built a new content management system to handle our publishing requirements.

Starting from the inside out, this is what our technology stack looks like.

The Hardware

We have eight HP servers, all with 8-core hyper-threading CPUs, and a minimum of 12 gigs of RAM. Four of these are at our Wellington office, and four are at our Auckland office. We'd previously hosted five machines at ICONZ data centre in Auckland.

Each group of machines is connected to the internet via a Juniper Firewall with dual edge-routers (for redundancy) connecting to FX Networks via Gigabit fibre. And yes, we can run both sites at 1 Gigabit if we need to.

Self-hosting was, as they say, a no-brainer. We have a server room at both locations, each with UPS and on-site generator. The deal we have with FX provides price certainty under a range of extreme scenarios. For example, the traffic we experienced after the Christchurch earthquake in February 2011 was 50 times higher than normal and stayed at 15% higher for days afterwards. We already move about 500 Gigabytes of data a day, so large peaks in traffic can cause large peaks in cost.

The dual site approach allows for us to continue operating if one site is cut off for some reason, and to share load between them if required.

We did consider hosting in the cloud, but two things counted against this approach. The first is we had eight high-end servers with 3 years depreciation still to run, and secondly the cost of provisioning for peaky and unpredictable traffic patterns. Our strong preference is to host in New Zealand to give local visitors the fastest possible service, and the price-point of local cloud services is not quite there for an operation of our size.

The Operating Systems

Our primary operating system (on 6 servers) is GNU/Debian Linux. All run as bare-metal with virtual machines (VM) running on top. VMs can be moved between cities with almost no gap in service.

We also have a pair of Windows Media Servers (one in each town) to handle live streaming and on-demand audio, but these will be retired later in the year and replaced with a Wowza streaming server. That change will give us much better cross-platform streaming capability.

The whole cluster was set up by Simon Blake of Katipo.

The Content Management System

Our content management system (CMS) is called ELF, and was built using the Ruby On Rails framework. We chose Rails because it is used internally for some of our intranet applications, and because it was a good fit for type of content we offer.

This replaces MySource Matrix, which we used from 2005 until we started phasing it out in 2010.

The first code for ELF was written on 8 March 2010, and in the month that followed Nigel Ramsay and Marcus Baguley from AbleTech wrote a proof on concept based around the recipes section of the site.

I was learning Rails myself at this point, from a background in 8086 Assembly, C, Perl and PHP. Later in the project we used external contractors for larger blocks of work and what I call 'heavy lifting' - complex features that require extended periods of work. Shevaun Coker has done much of this work and was joined by Cameron Prebble recently (both from AbleTech).

A proof of concept was used to assess the search tool (Apache Solr) and load profile of the system (how many pages it could serve). This was successful, with the platform being able to serve around 850 requests a second. This compared very well to the 20-30 requests a second upper limit of the old system.

Over the following 18 months large sections of the site were replaced, with news (the largest) replaced just weeks before the 22 February earthquake.

The last major section of Matrix was replaced late last year. In all, several hundred thousand pieces of content were migrated from Matrix to ELF.

The administration section of ELF has been written specifically to cater to the needs of busy broadcast producers and is highly optimised for the work that we do. We have built only the features that we need, avoiding clutter from features we don't want or use.

A producer can learn all they need to know about publishing their content in about 10 minutes.

ELF is about 126,000 lines of code, and the test suite has 2,200 tests that help us ensure the system remains free of bugs. A recent check of the system found that on average vistors to the site got a system error (a 500 error page) once every 40,000 page views. All bugs get attended to, so that number is now even lower.

We use a Varnish cache in front of the Rails application server, and they work in concert to deliver pages as quickly as possible, even when there is high demand.

There is a very detailed series of posts about the design and building of ELF, and the migration of content, right here on this blog. The series is called Rebuilding Radio NZ.

Publishing Systems

We publish content in three main ways:

• manually, via the administration interface of ELF
• from our news editing system (iNews);
• and from our on-air audio system (CoSTAR)

I have written software that allows our text and audio system to publish directly into the CMS. This enables staff to use their regular tools to publish, and no knowledge of HTML is required.

News content takes 15 seconds to update once the publish key is pressed, while audio takes a little longer due to the need to encode it for web delivery. We typically can have a piece of audio on the web a few minutes after broadcast.

Pre-recorded programmes (such as Our Changing World, Insight, Spectrum and many others) are pre-published, with content going live automatically after broadcast.

All these processes have been automated wherever possible, and they are highly optimised for speed, flexibility and reliability.

The Design

The aim was to simplify the site and let the content shine through.

We have retained many familiar elements in the design, improved others, and thrown away stuff that did not work. In the last three months there has been a vast amount of polishing work done - selecting great images, taking new staff photos, and reviewing all the content. It's been a long process!

Two years ago I wrote a post about dealing with doubt, and I wondered if we'd ever finish the project. I think if I'd known it would take another two years I might have given up then. Part of the problem was that a number of large projects I was working on got delayed, and I ended up working on them all at the same time.

Now we're done with this phase of the project, I am more than satisfied that the journey has been worth the trouble. We have a great new design, a CMS that'll cope with all our needs, and can continue to grow and be modified as we evolve the site.

The next phase is....

...you'll have to wait and see!

The Credits

My colleagues Frances Hopkins and Helena Nimmo deserve special mention. Their commitment and drive has ensured that the finished site has reached a level that is much, much more that the sum of its parts.

We've been supported by a great group of contractors (in no particular order): Shevaun Coker, Nigel Ramsay, Cameron Prebble, Marcus Baguley, Michael Koziarski, Alison Green, Susannah 'Roxy' Field, David Buck, Simon Blake, Amnon Ben-Or, Amanda Dorrell, John Moore, Phillipa Devenport-Johns, and Emma Harrison.

Feedback or comment about the the site can be sent to rnzwebsite@radionz.co.nz.

I am happy to answer any technical questions in the comments.

Radio NZ stats February 2013

As anyone who hasn't been living under a rock for the last few years will know, there has been an explosion of the use of mobile computing devices over the last few years. At Radio New Zealand we've seen the percentage of mobiles devices accessing our website increase dramatically in the last 12 months.

One year ago the percentage of our traffic from mobile devices (including tablets) was 13%. This month it is 27%.

The type of devices has also shifted. A year ago iDevices and Android were neck and neck. This month devices powered by Android are the clear leader:

OS	Perc.
Android	59.77
iOS	38.79
Windows Phone	0.58
Blackberry	0.49
Symbian	0.15
Nokia	0.07
Samsung	0.05
Windows	0.05
Sony	0.04

I expect this trend to continue, but it is impossible to say how much further it'll go.

Browser use is also shifting. A year ago IE was at 40% and Firefox was at 21%. Safari was 16% and Chrome 12.76. The Android browser was 6%.

This month's figures:

Browser	Perc.
IE	30
Safari	17.4
Firefox	16.7
Chrome	16.3
Android Browser	15.6
Safari (in app)	2
Opera	0.57
Opera Mini	0.31

There has also been a big change in operating systems:

OS	12 months ago	Last month
Windows	69.7%	57%
Macintosh	14.9%	12.5%
Android	6.19%	17.5%
iPhone/iPad	6.4%	10.9%
Linux	1.37%	1.24%
Windows Phone	0	0.16%

If you are are still designing sites for one browser or OS, now might be the time to change your strategy.

Remembering Paul

I worked with Paul Holmes in the 1980s while he was the host of the 9am to midday talkback slot on 2ZB in Wellington.

I was a studio operator at the time - we played the commercials, took in feeds of news and other events and balanced the audio levels of the show. I also got to worked with Paul on the odd outside broadcast, and helped put together the audio package that was part of his (first) winning entry in the Mobil Radio Awards. He offered me a lift up to Hawkes Bay as repayment - his Mum lived there, as did friends of mine - but I never collected. He always appreciated the work you did for him.

One of my most enduring memories is of his daily reading of an 'episode' - an often torrid section of a Mills & Bon novel - dressed up with an old-time music intro and outro and read in the most over-the-top fashion. Paul had been an actor, a contemporary of Sam Neil I think and also John Clarke, and he could really put on a character. That segment was very popular with the listeners, and I would often ask at the start of each shift if he was going to do an 'episode' today.

On-air, Paul was a stand-out. Talk-back has every type of caller from the nutty-conspiracy-theorists through to the lonely and alone. Paul could handle anything.  He was quick-witted, and could always come-back to anything thrown at him. But he was also compassionate, and I believed he cared deeply about the callers who phoned to talk about their troubles.

I remember him looking through the glass (soundproofing between the control room and his booth) on many occasions with his mischievous smile, looking for encouragement, wanting the producer and I to egg him on, which of course we did!

It was exciting working with Paul. You never knew what was going to happen, what direction today's show would go, whether this time he'd go a little too far. I don't recall him getting into big trouble while at 2ZB, but everyone knows some of the more famous (or infamous) incidents that happened later. I think the very best broadcasters are always on that fine-line between seat-of-the-pants and a train-wreck, and I Paul was a master at walking that line.

On a technical note, on one occasion he agreed that I could try out a new theory I had for better balancing the volumes of the ads, the phone audio and him. It was quite successful and the findings formed the basis of a report I wrote later for TVNZ about how to fix their loud commercial problems. (They never did action it).

The rebranding of 1ZB in Auckland to a NewsTalk (in 1987) had been a big secret, and quite a few people we in shock when it was announced that Paul was moving to Auckland. We knew that Merv Smith wasn't that happy with the format change and as it turned out his loyal Auckland audience wasn't either. The show dropped to number two, but eventually bounced back. The format was extended to 2ZB and 3ZB in the early 90s, having been proven in the tough Auckland Market. 

I did watch Holmes in it's early days, and could see many of the same attributes I'd seen working with him in Radio - his wit, and his compassion. You could sometimes see he was looking at the floor crew, presumably seeking feedback, as he'd done on radio. It was a great show. He really did care about the little guy, and this was borne out later when his involvement in community groups became more widely known.

Of course he left TVNZ for Prime, after contract negotiations went bad - I think TVNZ was offering only a one year renewal. Viewers did not follow, and this was attributed to the market inertia that TV1 had - for many people it was the only channel they had ever watched, and I believe that Paul contributed to maintaining that inertia, sadly to his own detriment when he moved on. It's shame the audience was too entrenched to follow him, for Paul thrived with an audience.

At the Radio Awards in the early 90s I walked past him, not thinking he'd remember me, and he asked, "me old mate, how are are you?". 

Years later (around 2000) I was walking up past the TVNZ building in Auckland. Paul was outside waiting to do a piece to camera. He looked over, saw me and raised his eyebrows and titled his head back slightly in acknowledgement, the way blokes do to each other. Even though we only worked together for a short time a long time before, for a second it was just him and me.

That's how I'll remember Paul.

A Weekend in Whanganui

Normally I write about tech issues here, but this last weekend my wife and I were tourists, for a get-away without the kids. A tourist's view of a location is important - in this on-line age word of mouth is probably the most important influencer of opinion about where to holiday - especially with review sites likes Trip Advisor.

With this in mind, this is my review of our weekend away in Whanganui.

First Impressions

We drove into town from State Highway 4, over the bridge and up Victoria Avenue. First impressions are important, and what a first impression! A lot of work has obviously been put into the tree-lined main street, and I found out later that local businesses pay a levy which funds various initiatives.

The mature trees, footpath paving, hanging baskets and wrought iron fittings create a cohesive and inviting feel, and during our stay we walked the length of Victoria Avenue several times. We enjoyed looking at the old buildings and reading about the history (where there were plaques). Some of these were are bit old and hard to read. Please update them!

Shopping

There was shopping. There is always shopping.

The thing that struck us was how universally friendly shop staff were. We were greeted when we went into shops, and thanked when we left. This is quite common in small towns, where tourists are obvious, but in a town the size of Whanganui this was a pleasant surprise.

There was a nice little gift shop near the information centre on Taupo Quay. Not over-priced like so many shops and pretty

The Galleries

Caroline is an artist, so we visit galleries. I am a willing participant in this and fortunately out tastes overlap.

We visited the Sarjeant Gallery and some smaller places dotted around the edges of Queens Park. I was not at all concerned by the earthquake notice at Sarjeant, but more concerned about what might happen to this amazing space.

Both of us enjoyed Joe Sheehan's stone work, and the showcase of "Artists working in Wanganui, 1910 - 2012".

Eating Out

One of the best parts of going away is eating out and our first meal out was at the Angora Cafe and Restaurant. Upon being shown to our table we asked to be moved because the main dining area was far too hot. We were reseated in an area that is open to the street. The waitress was very good, and the food excellent.

Two things detracted from the visit. The first was a girl working behind the counter wearing sports shorts, and with white ear-buds on, in direct view from where I was sitting. When paying, I didn't appreciate seeing the same person preparing deserts, and rearranging the food with her fingers. Call me old-fashioned, but I want my food prepared in the kitchen by professionals.

The second was the poor attitude of the woman on the desk. No "thank you for eating here, did you enjoy the meal". When my wife mentioned that there was some shards of broken glass by our table, rather than apologising, she said "it must of been from yesterday". This is a shame because a local recommended we eat there, and as I said the food was excellent.

Our second evening meal was at Element Cafe and Restaurant, situated in an old bank building, and having an outside dining area. 

As it happens, we also had lunch here the day before. Both times we were looking at the menu at the door and a waitress came up to explain the different options. Both were very friendly, but not pushy, and in both cases this swayed us to stay.

The food was excellent, and I want to note the freshness of the fish-of-the-day (snapper) and the Blackcurrant, Black Dorris & Marzipan Pie I had for desert. My wife had the Caesar Salad, and the egg was cooked perfectly, with very fresh salad greens. The service was relaxed but attentive. We moved between our main and desert because our street site-side table was getting a little cool. Not a problem. We ordered mains off the cafe menu and desert off the bistro menu. Not a problem. We move inside for our coffee. Not a problem (and our waitress raced over to wipe down the table in the lounge area because of one small piece of cream). Going back? Definitely.

Bushy Park

Bushy Park must be one of the North-Island's best kept secrets. The manager of the Park's homestead (above) told me that few locals visit, and there were about 5 cars there when we visited on Sunday, their busiest day.

The park itself is a 245 acre remnant of low-land native rainforest, surrounded by a predator-proof fence and managed by a trust. The story of the estate is fascinating, and well worth reading.

Access is drive-in via a series of special gates, and there are a range of walks. If we'd had more time, we would have done them all. The forest is so peaceful, apart from the call of native birds. If you are quiet, the birds seem to find you, and the Robin (at right, taken by Caroline) was just a few metres away and seemed quite inquisitive.

As a size comparison, the park is just over half the size of Wellington's Zealandia. Entry is $6 a head, kids for free. They have basic lunch available, made on-demand; we had a toasted sandwich, chips, and devonshire tea, served with silver tea-ware and china, taken on the front veranda of the homestead.

If you are in Whanganui, as a resident or visitor, this is a must see.

They also operate a bed-and-breakfast where you can stay in the homestead's historic rooms.

The Steamer

We were on our way out of town at 10:45 am and saw the river steamer smoking up. We'd wanted to go, but missed out on Saturday.

During the trip an on-board speaker system gave us information about local landmarks and history, with occasional live announcements. The drinks and scones we had were very reasonably priced for an attraction of this type - so often prices go up because of the captive audience. And the scones were very fresh. "Come back for more jam, or cream, if you need it", we were told. I really appreciate small touches like this.

We had a look in the boiler room, where were we told about the boiler and steam engines (two at 86 horse power each with lots of torque), and how they maintain pressure - it's a mix of air control and coal.

A very positive end to an enjoyable weekend.

Other Highlights

We had a chat with one of the volunteers at the Tram Shed about the history of trams in Whanganui, and their plans for future developments. I was astounded to find that the building is only a few years old, built from mostly recycled building parts. From the outside it looks 100 years-old. It was built with PD (periodic detention) labour, and we were told that some of the crew who worked on it still visit to see their handy-work. 

We stayed at the Sienna Motor Lodge and the rooms were outstanding. (Because of us booking so late we changed rooms once, by choice, to get the facilities we wanted). The rooms were spotless. Really spotless. I have stayed in many motels and hotels and the rooms were the among the cleanest I have stayed in. It is inevitable that you find things broken or not working in motel rooms - it must be near-impossible to keep on top of maintenance - but not in this case. On the last morning we ordered breakfast. Motel breakfasts can be indifferent, but in this case I wish I had taken a photo as a lot of care had been put into cooking and presenting what was a simple meal for two. 

We will certainly be going back to Whanganui.

How Not To Do Customer Support (updated)

Update: I've heard from McDonalds - scroll down.

While in Whanganui recently (a full review to come), we popped in to McDonalds in Victoria Avenue. I was hankering for a Grand Angus burger.

I went to the counter and was told I would be served in a minute. While I waited my wife went to sit down. After a few minutes she came back to say that she'd cleared a table but that it was too hot inside. I suggested finding a table outside.

After a few more minutes she came back to say the tables outside were pilled with rubbish, and she'd also noticed that all the bins were overflowing.

Standing at the counter (still waiting for service) I could see other signs that all was not well. There were frozen chips and a bag on the floor by the chip frier. Quite a few people walked past, but no-one cleaned them up.

The kitchen was mostly silent. No one seemed to be communicating, and if you know anything about commercial kitchens, a silent kitchen is a kitchen in trouble.

On the counter were three order tickets from the till, the only till in use at 6:30 pm! A server called out an order and a man got up from a seat to collect it. Looking outside, I could see three cars waiting in the drive-through over-flow parks.

The chip area had a huge pile of chips in the warming area. Given the size of the pile I suspect that they batch cooked a large amount of fries and used them as orders came in.

At this point we decided to leave, not having placed an order, and feeling pretty mad.

I called them when I got back to the motel to complain. The duty manager said that she was sorry. That was it. No, "if you'd like to come back...", no, "we'll sort those issues out", nothing.

By way of contrast our local McDonalds in Lower Hutt, is very fast, even at busy times, the staff are friendly, and generally speaking the burgers looks like someone actually cared when they were assembled. On the one occasion I had a complaint the franchise owner called me and thanked me for the valuable feedback, offering some vouchers. I felt guilty taking the vouchers because of the positive way she responded to our complaint. That's how you buy loyalty.

Anyway, I was so annoyed I took to twitter. I got a response to ring the CS team on a 09 number, which I did today. The person on the phone took the details of the store, and noted my issues with the visit, and then came the clincher. Would I like the franchise owner to contact me about the complaint?

I'd already made it clear that I know how a MD kitchen is supposed to work ( I know a number of people who've worked there). I said that the place didn't appear to run as an MD is supposed to. I was told that as this was a franchise it was up to the franchisee to respond.

I was speechless. They'd already had a chance to respond.

The issue here is not whether this is a franchise or not. This is National  Brand. A customer should be able to expect a similar experience regardless of where they are. If I was the owner of a national brand, I would be extremely concerned about a report of a store (apparently) operating in a way that eroded the brand values. I am not expecting a fine-dining experience, just something that matches what I've been used to elsewhere.

McDonalds have now had three opportunities to repair this relationship, and have not done so.

What could have been done differently?

Firstly, the local duty manager should have done more. Not dealing with the issue allowed the complaint to fester.

Secondly, if you use social media don't make me make a toll call (this one took me nearly 10 minutes), give me an 0800 number or better still get someone to call - "Please DM us your phone number and a time to call". Take control of the situation, don't make me do all the work (and pay for the call). Complaints are valuable feedback. Offer me a voucher for the trouble of calling.

Thirdly, I don't care who actually owns the place - it is a McDonalds. I already tried complaining on-site. If someone has already complained directly, complained on Twitter, and gone to the trouble of making a toll call, you can be sure that they are annoyed. Don't make me jump through more hoops.

Footnote: I mentioned our experience to a couple of locals - they both avoided the place because of recent bad experiences. I'd still use Lower Hutt McDonalds, but I'm going to be wary of others from now on. And least I did get a response. I don't ever go to KFC now after their Head Office failed to respond at all to a food safety complaint.

Update (10 Jan 2013)

I've just had a call from Kim, McDonalds' communications manager. They've taken on board my criticisms of the complaints process, and particularly how things were handled on Twitter. I appreciated having the chance to expand on a couple of the issues in this post, and hearing about their internal processes for checking that quality standards are maintained in their restaurants.

I understand that the local owner will be looking into what happened that night, as this is not typical for that store, and they don't have a history of complaints either.

So overall, I am happy with the outcome.

Is there a key lesson? Yes - customers want to be heard, and need to know that they have been heard. Complaints should be dealt with as soon as possible to avoid escalations.

Some general thoughts on this:

Front-line staff should be empowered (and trained) to handle complaints. In this case if the duty manager had said they were under-staffed due to illness, or explained some other mitigating factor (power cut, airconditioning failure resulting in sub-optimal performance)  I would have moved on; every store has bad days, and most customers will understand that.

The easiest way to do this is practice complaint scenarios in advance. Don't script them, but give staff the chance to free-form, giving them feedback on what worked and what did not. Practice helps staff build up the confidence they'll need dealing with an upset customer.

If a complaint is made via social media, respond as quickly as possible. On-lookers will see that you are responsive. Facebook allows for long comments, so it is easy to deal with complaints entirely on-line.

Twitter is more difficult, and my general suggestion is take it as far as you can on Twitter and then contact the person directly. It is easy for misunderstandings to occur due to the character limit. You can follow up on Twitter later, although most people who complain via Twitter are going to post something themselves later. In New Zealand I have seem positive follow-up Tweets after complaints made to Air New Zealand and Telecom, so this can work in you favour.

If you have an 0800 number, please don't make your staff follow a script. I take (as does my whole web team) complaints from users of my company's website. There are certainly questions that we need to ask every caller, but they can be weaved naturally into the conversation. Allow the other person to talk and ask questions to draw out the information you need. Not everyone will be coherent and forth-coming with information, and not everyone will describe things in the way you expect.

Disclosure: McDonalds is sending out a voucher for my trouble. It'll be used to get a Grand Angus, which I still think is the best fast-food burger out there. :-)