A Weekend in Whanganui

Normally I write about tech issues here, but this last weekend my wife and I were tourists, for a get-away without the kids. A tourist's view of a location is important - in this on-line age word of mouth is probably the most important influencer of opinion about where to holiday - especially with review sites likes Trip Advisor.

With this in mind, this is my review of our weekend away in Whanganui.

First Impressions

We drove into town from State Highway 4, over the bridge and up Victoria Avenue. First impressions are important, and what a first impression! A lot of work has obviously been put into the tree-lined main street, and I found out later that local businesses pay a levy which funds various initiatives.

The mature trees, footpath paving, hanging baskets and wrought iron fittings create a cohesive and inviting feel, and during our stay we walked the length of Victoria Avenue several times. We enjoyed looking at the old buildings and reading about the history (where there were plaques). Some of these were are bit old and hard to read. Please update them!

Shopping

There was shopping. There is always shopping.

The thing that struck us was how universally friendly shop staff were. We were greeted when we went into shops, and thanked when we left. This is quite common in small towns, where tourists are obvious, but in a town the size of Whanganui this was a pleasant surprise.

There was a nice little gift shop near the information centre on Taupo Quay. Not over-priced like so many shops and pretty

The Galleries

Caroline is an artist, so we visit galleries. I am a willing participant in this and fortunately out tastes overlap.

We visited the Sarjeant Gallery and some smaller places dotted around the edges of Queens Park. I was not at all concerned by the earthquake notice at Sarjeant, but more concerned about what might happen to this amazing space.

Both of us enjoyed Joe Sheehan's stone work, and the showcase of "Artists working in Wanganui, 1910 - 2012".

Eating Out

One of the best parts of going away is eating out and our first meal out was at the Angora Cafe and Restaurant. Upon being shown to our table we asked to be moved because the main dining area was far too hot. We were reseated in an area that is open to the street. The waitress was very good, and the food excellent.

Two things detracted from the visit. The first was a girl working behind the counter wearing sports shorts, and with white ear-buds on, in direct view from where I was sitting. When paying, I didn't appreciate seeing the same person preparing deserts, and rearranging the food with her fingers. Call me old-fashioned, but I want my food prepared in the kitchen by professionals.

The second was the poor attitude of the woman on the desk. No "thank you for eating here, did you enjoy the meal". When my wife mentioned that there was some shards of broken glass by our table, rather than apologising, she said "it must of been from yesterday". This is a shame because a local recommended we eat there, and as I said the food was excellent.

Our second evening meal was at Element Cafe and Restaurant, situated in an old bank building, and having an outside dining area. 

As it happens, we also had lunch here the day before. Both times we were looking at the menu at the door and a waitress came up to explain the different options. Both were very friendly, but not pushy, and in both cases this swayed us to stay.

The food was excellent, and I want to note the freshness of the fish-of-the-day (snapper) and the Blackcurrant, Black Dorris & Marzipan Pie I had for desert. My wife had the Caesar Salad, and the egg was cooked perfectly, with very fresh salad greens. The service was relaxed but attentive. We moved between our main and desert because our street site-side table was getting a little cool. Not a problem. We ordered mains off the cafe menu and desert off the bistro menu. Not a problem. We move inside for our coffee. Not a problem (and our waitress raced over to wipe down the table in the lounge area because of one small piece of cream). Going back? Definitely.

Bushy Park

Bushy Park must be one of the North-Island's best kept secrets. The manager of the Park's homestead (above) told me that few locals visit, and there were about 5 cars there when we visited on Sunday, their busiest day.

The park itself is a 245 acre remnant of low-land native rainforest, surrounded by a predator-proof fence and managed by a trust. The story of the estate is fascinating, and well worth reading.

Access is drive-in via a series of special gates, and there are a range of walks. If we'd had more time, we would have done them all. The forest is so peaceful, apart from the call of native birds. If you are quiet, the birds seem to find you, and the Robin (at right, taken by Caroline) was just a few metres away and seemed quite inquisitive.

As a size comparison, the park is just over half the size of Wellington's Zealandia. Entry is $6 a head, kids for free. They have basic lunch available, made on-demand; we had a toasted sandwich, chips, and devonshire tea, served with silver tea-ware and china, taken on the front veranda of the homestead.

If you are in Whanganui, as a resident or visitor, this is a must see.

They also operate a bed-and-breakfast where you can stay in the homestead's historic rooms.

The Steamer

We were on our way out of town at 10:45 am and saw the river steamer smoking up. We'd wanted to go, but missed out on Saturday.

During the trip an on-board speaker system gave us information about local landmarks and history, with occasional live announcements. The drinks and scones we had were very reasonably priced for an attraction of this type - so often prices go up because of the captive audience. And the scones were very fresh. "Come back for more jam, or cream, if you need it", we were told. I really appreciate small touches like this.

We had a look in the boiler room, where were we told about the boiler and steam engines (two at 86 horse power each with lots of torque), and how they maintain pressure - it's a mix of air control and coal.

A very positive end to an enjoyable weekend.

Other Highlights

We had a chat with one of the volunteers at the Tram Shed about the history of trams in Whanganui, and their plans for future developments. I was astounded to find that the building is only a few years old, built from mostly recycled building parts. From the outside it looks 100 years-old. It was built with PD (periodic detention) labour, and we were told that some of the crew who worked on it still visit to see their handy-work. 

We stayed at the Sienna Motor Lodge and the rooms were outstanding. (Because of us booking so late we changed rooms once, by choice, to get the facilities we wanted). The rooms were spotless. Really spotless. I have stayed in many motels and hotels and the rooms were the among the cleanest I have stayed in. It is inevitable that you find things broken or not working in motel rooms - it must be near-impossible to keep on top of maintenance - but not in this case. On the last morning we ordered breakfast. Motel breakfasts can be indifferent, but in this case I wish I had taken a photo as a lot of care had been put into cooking and presenting what was a simple meal for two. 

We will certainly be going back to Whanganui.

How Not To Do Customer Support (updated)

Update: I've heard from McDonalds - scroll down.

While in Whanganui recently (a full review to come), we popped in to McDonalds in Victoria Avenue. I was hankering for a Grand Angus burger.

I went to the counter and was told I would be served in a minute. While I waited my wife went to sit down. After a few minutes she came back to say that she'd cleared a table but that it was too hot inside. I suggested finding a table outside.

After a few more minutes she came back to say the tables outside were pilled with rubbish, and she'd also noticed that all the bins were overflowing.

Standing at the counter (still waiting for service) I could see other signs that all was not well. There were frozen chips and a bag on the floor by the chip frier. Quite a few people walked past, but no-one cleaned them up.

The kitchen was mostly silent. No one seemed to be communicating, and if you know anything about commercial kitchens, a silent kitchen is a kitchen in trouble.

On the counter were three order tickets from the till, the only till in use at 6:30 pm! A server called out an order and a man got up from a seat to collect it. Looking outside, I could see three cars waiting in the drive-through over-flow parks.

The chip area had a huge pile of chips in the warming area. Given the size of the pile I suspect that they batch cooked a large amount of fries and used them as orders came in.

At this point we decided to leave, not having placed an order, and feeling pretty mad.

I called them when I got back to the motel to complain. The duty manager said that she was sorry. That was it. No, "if you'd like to come back...", no, "we'll sort those issues out", nothing.

By way of contrast our local McDonalds in Lower Hutt, is very fast, even at busy times, the staff are friendly, and generally speaking the burgers looks like someone actually cared when they were assembled. On the one occasion I had a complaint the franchise owner called me and thanked me for the valuable feedback, offering some vouchers. I felt guilty taking the vouchers because of the positive way she responded to our complaint. That's how you buy loyalty.

Anyway, I was so annoyed I took to twitter. I got a response to ring the CS team on a 09 number, which I did today. The person on the phone took the details of the store, and noted my issues with the visit, and then came the clincher. Would I like the franchise owner to contact me about the complaint?

I'd already made it clear that I know how a MD kitchen is supposed to work ( I know a number of people who've worked there). I said that the place didn't appear to run as an MD is supposed to. I was told that as this was a franchise it was up to the franchisee to respond.

I was speechless. They'd already had a chance to respond.

The issue here is not whether this is a franchise or not. This is National  Brand. A customer should be able to expect a similar experience regardless of where they are. If I was the owner of a national brand, I would be extremely concerned about a report of a store (apparently) operating in a way that eroded the brand values. I am not expecting a fine-dining experience, just something that matches what I've been used to elsewhere.

McDonalds have now had three opportunities to repair this relationship, and have not done so.

What could have been done differently?

Firstly, the local duty manager should have done more. Not dealing with the issue allowed the complaint to fester.

Secondly, if you use social media don't make me make a toll call (this one took me nearly 10 minutes), give me an 0800 number or better still get someone to call - "Please DM us your phone number and a time to call". Take control of the situation, don't make me do all the work (and pay for the call). Complaints are valuable feedback. Offer me a voucher for the trouble of calling.

Thirdly, I don't care who actually owns the place - it is a McDonalds. I already tried complaining on-site. If someone has already complained directly, complained on Twitter, and gone to the trouble of making a toll call, you can be sure that they are annoyed. Don't make me jump through more hoops.

Footnote: I mentioned our experience to a couple of locals - they both avoided the place because of recent bad experiences. I'd still use Lower Hutt McDonalds, but I'm going to be wary of others from now on. And least I did get a response. I don't ever go to KFC now after their Head Office failed to respond at all to a food safety complaint.

Update (10 Jan 2013)

I've just had a call from Kim, McDonalds' communications manager. They've taken on board my criticisms of the complaints process, and particularly how things were handled on Twitter. I appreciated having the chance to expand on a couple of the issues in this post, and hearing about their internal processes for checking that quality standards are maintained in their restaurants.

I understand that the local owner will be looking into what happened that night, as this is not typical for that store, and they don't have a history of complaints either.

So overall, I am happy with the outcome.

Is there a key lesson? Yes - customers want to be heard, and need to know that they have been heard. Complaints should be dealt with as soon as possible to avoid escalations.

Some general thoughts on this:

Front-line staff should be empowered (and trained) to handle complaints. In this case if the duty manager had said they were under-staffed due to illness, or explained some other mitigating factor (power cut, airconditioning failure resulting in sub-optimal performance)  I would have moved on; every store has bad days, and most customers will understand that.

The easiest way to do this is practice complaint scenarios in advance. Don't script them, but give staff the chance to free-form, giving them feedback on what worked and what did not. Practice helps staff build up the confidence they'll need dealing with an upset customer.

If a complaint is made via social media, respond as quickly as possible. On-lookers will see that you are responsive. Facebook allows for long comments, so it is easy to deal with complaints entirely on-line.

Twitter is more difficult, and my general suggestion is take it as far as you can on Twitter and then contact the person directly. It is easy for misunderstandings to occur due to the character limit. You can follow up on Twitter later, although most people who complain via Twitter are going to post something themselves later. In New Zealand I have seem positive follow-up Tweets after complaints made to Air New Zealand and Telecom, so this can work in you favour.

If you have an 0800 number, please don't make your staff follow a script. I take (as does my whole web team) complaints from users of my company's website. There are certainly questions that we need to ask every caller, but they can be weaved naturally into the conversation. Allow the other person to talk and ask questions to draw out the information you need. Not everyone will be coherent and forth-coming with information, and not everyone will describe things in the way you expect.

Disclosure: McDonalds is sending out a voucher for my trouble. It'll be used to get a Grand Angus, which I still think is the best fast-food burger out there. :-)

Radio NZ's New Hosting

Since 2005 Radio NZ has hosted it's page server at ICONZ's hosting facility in Auckland. We also had a connection in Wellington to allow us to publish content to the live system via a VPN.

Also, since 2005, we've used Citylink for audio content delivery. Their CDN, initially run as a side project by Richard Naylor, was the only economic way for us to deliver our audio streaming and downloads.

Until this year.

The Background

We rely on computer-based audio tools, and as a life-line utility we also need to have resilient infrastructure including full UPS and generator back-up for our main offices.

So self-hosting was always possible, but the cost of bandwidth to our premises made it uneconomic. In late 2004 we were quoted $87,000 a month for a 1 Gig internet connection.

As late as 2010 we were still being quoted prices of $4,000 per month for local connectivity to all NZ consumers. Fibre connectivity and International data was on top of that!

Earlier this year we issued an RFP for gigabit fibre and bandwidth from our Wellington and Auckland offices. We became an APNIC member and have our own IP addresses and AS number, enabling us to participate directly in the internet at a technical level.

FX Networks won the tender process with an innovative data plan and is now the ISP for all our public facing web services. These includes our website, the live streaming service and our podcasts.

Because of FX's extensive peering arrangements, all traffic is delivered from New Zealand. For customers of two large national ISPs, their content will no longer have to be imported from a server in the U.S.

International visitors make up between five and 15% of traffic (time dependant) and they'll get the traffic from NZ as well. This means for the vast majority of visitors latency will reduce, and the speed of downloads may go up.

The Migration

The major constraint on the migration was downtime - there was to be none. The second was that we could not buy any more servers, having just bought eight just three years ago, meaning we had to work out a way to run the existing site at ICONZ and a syncronised clone of it from one of our offices via FX.

As it happens our old CMS (MySource Matrix) required 6 servers, and we had two originating servers for the Citylink CDN (one, a live spare). But we have since replaced our CMS with something that needed fewer resources, freeing up some servers to allow us to build the new system in tandem with the old.

So far, so good.

The new infrastructure uses virtual machines (VM), running on GNU/Linux (except for Windows Media Server). All VMs must be able to run in Auckland or Wellington to allow for services to continue if connectivity to one office is lost (such as in a disaster).

Each VM has a static IP address that is used at either location, and a combination of Routing Information Protocol (RIP) and Border Gateway Protocol (BGP) are used to ensure requests go to the right place within our network.

With two servers relocated to our Wellington office and two to Auckland, a basic cluster was built. At each site dual edge routers connect to FX Networks over fibre, and an SRX240 handles firewall duties. A number of VPN tunnels run between the sites to allow disc replication and communication between VMs in the cluster.

The Backup media and podcast server was moved from the ICONZ address space to the FX connection and prepped to take over from the CDN.

One the cluster bare-metal hardware was in place, the existing Web servers were cloned into new VMs, and reconfigured to work within the new environment.

The audio service change-over was the simplest, requiring a DNS change. The DNS records for streaming.radionz.co.nz and podcast.radionz.co.nz were updated to point to our server, and over the space of a day or so all load moved from Citylink's CDN to our server in Wellington. The second server will be moved from ICONZ to our Auckland office, to act as a live spare.

The website change-over was more complicated. Because we are publishing so frequently, we could not do a straight DNS change because some people would be getting the old (no longer updated site) for a period of time. There were several ways to keep the sites in sync, but given the complexity (we are still using parts of MySource Matrix), it wasn't worth the pain to cover the one- to eight-hour DNS time. (And before you ask, not all ISPs honour a short TTL).

The option I chose was to setup a Varnish caching server, point that at the current site, and change the DNS to direct traffic to Varnish. As request moved from the existing site to Varnish, they were forwarded silently back to the old. Once all the traffic requests were arriving at the Varnish, we could then syncronise the new VM servers with the old live servers, and update Varnish to fetch from the those.

This approach meant we could synchronise the new servers with the old (which took about 30 minutes), and then instantly switch Varnish to send requests to the new machines. This we did at 8:45 pm last night. During the syncing process our newsroom could still publish updates to the old server, and the first publish on the new server brought everything up to date.

It is interesting to note that we started directing traffic to the Varnish host just before a tornado hit Auckland, and during this period we saw four times the usual amount of traffic to the site. All of this was going via Varnish, and there was not a single outage.

We will soon move the remaining servers out of ICONZ and add them to the new cluster, improving redundancy and load-balancing capabilities further. From there we'll be fine-tuning the overall performance of the cluster, and next year launching a new design.

I'm happy to answer any technical question in the comments.

Latest Stats for Radio NZ

I periodically publish stats on browser and OS use at www.radionz.co.nz, and this time I'm going to include some content and mobile stats.

Size of Content Library

The audio library now contains 17,000 hours of material, all of it searchable from www.radionz.co.nz/audio. Most programmes go back to 2008, some (Our Changing World - 2005 and Spectrum - 2001) go back further.

We are adding to this library at the rate of 13.5 hours a day. Almost everything is downloadable and embeddable.

Browsers

Browser	06/2012	11/2011	11/2010	11/2009	11/2008
IE	37.5	41.2	50.6	56	63
Firefox	19.8	23.2	25.52	27.5	27.73
Safari	17.3	5.6	13.1	10	5.66
Chrome	14.6	13.8	8.75	4.2	1.47
Android	7.5
Opera	0.69	0.7	0.9	1.02	1.08

IE is in decline, and IE6 is currently 3.25% of IE browser share.

Operating System

OS	06/2012	11/2011	11/2010	11/2009	11/2008
Windows	67.3	72	81	84.8	89.3
Mac	14.7	15.6	14.2	12.6	8.5
iOS	7.88
Android	7.79	3.6	0.3	0.02	0
iPhone		2.5	1.4	0.56	0.19
iPad		2.4	0.63	0	0
Linux	1.27	1.53	1.4	1.45	1.72
iPod		0.5	0.35	0.22	0.08

Mobile

The number of people visiting from mobile is high. 16% in fact.
Here is a breakdown of the top ten devices.

Device	%	Average visit duration	Pages/visit
iPad	25.5	2:13	2.3
iPhone	21.0	1:52	1.7
not set	11.12	1:14	1.56
Samsung GT-I9100 Galaxy S II	6.85	0:54	1.43
Huawei U8150 Ideos	3.36	0.53	1.26
Samsung GT-S5570 Galaxy Mini	3.34	0:37	1.27
SonyEricsson LT15i Xperia Arc	3.1	1:00	1.51
iPod Touch	3.01	3:42	2.26
Samsung GT-S5830 Galaxy Ace	2.73	0:41	1.25
Motorola MB525 DEFY	2.73	0:48	1.42

Site Speed

We have shaved nearly a second off our Google site speed in the last 12 months, although we are still being hampered by having two CMSs running different parts of the site. I suspect this is one reason for the high mobile use of the site - it is fast on these devices.

The Homeopathic Hot Water Bottle

I am departing from my usual posts about technology, to share a remarkable discovery.

For years I have been skeptical about the claims of homeopathy - that minute amounts of a substance can have therapeutic properties. In the last few weeks I have changed my mind after finding proof that anyone can try for themselves. And the discovery was made completely by chance!

Everyone knows that a hot water bottle (HWB) gets cold at a certain (fixed) rate. The rate is dependent on the amount of bedding and any other factors which determine heat-loss. One night I recycled the cold water from the night before, and a family member commented that the HWB was still warm in the morning. This seemed odd, so I investigated further. 


What I found is that the rate of heat loss can be altered using homeopathic techniques. Here is how.

On the first night make the water for your bottle with 400mL of cold tap-water and 800 mL of boiling water. This mix gives a temperature which is safe - hotter than this can cause the bottle to perish more quickly and leak. Extensive scientific testing has also found this temperature to about right for certain family members.

On the second night use 400mL of cold water taken from the bottle, adding 800 mL of fresh boiling water. Repeat each night for a week.

You will find each night the HWB retains it heat better. The effect seems to flatten out after a week.

The only explanation I have for this is that the recycled cold water carries the memory of being hot, and for some reason, when exposed to fresh hot water, prefers to stay hot.

Amazing (and useful)!