How to Fix Loud Commercials on TV - Part 1

This is the first of a series of non-tech posts about the volume of commercials on TV - why they are too loud and how to stop the problem.

The problem of commercials being too loud is the result of an arms race, of sorts, that has its roots in practices that were set decades ago. But it is not a race to the top, to world domination, to commercial success. It is a race to the bottom, to the lowest possible quality and the worst outcome for all.

The problem exists world-wide. The US have passed a law, and in New Zealand it is Labour Party Policy. Debates rage on forums, both public, amateur and professional about the cause of the problem and what can be done about it.

About Me

Before running Radio NZ's web operation I was a recording engineer. I started out as a Trainee Radio Studio Operator in 1981, and I've worked in commercial and public radio, on sports broadcasts, and music recording of all genres from early music through the classics, world music, jazz and rock. I've also recorded film scores, and mastered and re-mastered albums.

In the 80s I set-up the audio processing for a couple of Wellington radio stations, 91ZM (now ZMFM) and 2ZB (now NewstalkZB). I was in the fortunate position of having made commercials, done on-air sound mixing (called panel operating in some countries) and worked on the station sound (via audio processing). This allowed me to try out my ideas for improvement on-air, and hear the results first-hand.

Sometime in the mid 80's I was asked to contribute to discussions at TVNZ about the problem of loud commercials.

The Problem

I will start by defining the problem from the viewer's (and listener's) perspective.

You turn on the television to watch a programme. You sit down, adjust the volume so that it is comfortable for you, and start to enjoy the programme. As the show moves from scene to scene (assuming there is no ad break) the volume remains comfortable; you can hear everything that is being said, any  music and effects are neither too loud or too quiet.

Then a commercial break arrives. The volume suddenly increases. It is no longer comfortable to listen to - it is intrusive. During the programme you could have a side-conversation with your fellow viewers. That is now impossible. You reach for the remote and mute the audio.

This is the experience of hundreds of millions of television viewers.

The perception is that someone, somewhere, is turning up the commercials.

So, the problem in a nutshell: the volume of broadcast items is inconsistent, to the point of being disruptive and annoying.


Fixing the Problem

It is very unlikely that anyone, anywhere, is turning up the commercials. I've certainly never seen it.

The problem is caused by a number of technical and operational factors, and is (probably) rounded off by management being unwilling to deal with the issues for 'competitive reasons'.

The problem is complex, and I want to make the explanation accessible to people outside the audio industry. I'll spend a few posts looking at various aspects of the problem, explaining some of the basics of audio, hearing and listening before moving onto solutions.

Please leave any questions, or things you want explained in the comments.

Latest Radio NZ Browser stats

The latest browser stats for www.radionz.co.nz show some interesting changes when compared to previous years.

Browser	2011	2010	2009	2008
IE	41.2	50.6	56	63
Firefox	23.2	25.52	27.5	27.73
Safari	5.6	13.1	10	5.66
Chrome	13.8	8.75	4.2	1.47
Opera	0.7	0.9	1.02	1.08

IE is in decline, and IE6 is currently 3.6% of total browser share. Operating system use is also changing.

OS	2011	2010	2009	2008
Windows	72	81	84.8	89.3
Mac	15.6	14.2	12.6	8.5
Android	3.6	0.3	0.02	0
iPhone	2.5	1.4	0.56	0.19
iPad	2.4	0.63	0	0
Linux	1.53	1.4	1.45	1.72
iPod	0.5	0.35	0.22	0.08

Windows use is dropping, and mobile operating systems are on the rise. Mac use in 2005 was less than 5%.

The breakdown of mobile devices share in the last month:

Android - 38
iPhone - 27
iPad - 26
iPod - 5.33
Blackberry - 0.8
Symbian - 0.7
Sony - 0.19
Windows - 0.12
Nokia - 0.11
Windows Phone - 0.1

iPad users spend on average twice as long on the site.

Over on Geekzone Mauricio has posted page speed info for his website.

Here is similar information for Radio New Zealand.

Average web page load time distribution in New Zealand:

All regions:

Slowest cities:

This is running on a server based at ICONZ running Rails 3.1.x.

The key thing we've done to make our site fast is reducing over-the-wire times by bundling assets, using compression where we can, and setting far-future headers for static content to ensure it can be cached somewhere else. The markup is also as clean as we can make it to ensure fast rendering.

Radio NZ's embedded player

Last week Radio New Zealand released its new embedded player to the general public.

The graphics were designed by Clemenger BBDO in Wellington and implemented in Flash by PixelDepth in Auckland. The player (and server software) took 3.5 person-days to code, implement and test.

The player can be used by anyone (subject to terms of use) to embed Radio NZ audio content on their web pages.

While some people will be disappointed that the player does not work on iOS devices, the decision to make this first version in Flash was carefully considered. The core web team at RNZ is small - there are just three of us working on a range of projects. At the moment we are working on a complete redesign of the site, and on replacing our CMS, not to mention planning web coverage for the upcoming election.

A few years ago I started work on an HTML5 based player for our audio. The code is still available on Github. One of the impediments to completing this project was the lack of time to debug the application. At the time browser support for HTML5 was patchy. Building that player gave me a good idea of the level of engineering required for a complete package. I take accessibility very seriously. Any player has to work on all platforms, and for screen readers.

I have started planning for release 2 of the embedded player, but it will need thorough testing before release. We didn't have the time to do that now. I'm sure you all understand!

The new CMS is providing the data-backend for the new player, and audio content is delivered from our content delivery network. Both of these are completely platform agnostic. The iframe technique we are using will allow the player to be updated in the future, and for this to appear everywhere.

The player is already being used more widely than I'd expected. Other public broadcasters like NPR, special interest sites like Treetools, and Cycling Wellington, blogs, not to mention local news websites.

Rebuilding Radio NZ - Part 12: Migrating Episodes

The migration of programme episode content from MySource Matrix into our new Rails-based CMS has proven to be the largest and most complex task. This week I am going to dive heavily into the code we used to do this.

For most programmes we have maintain a programme library of content dating back to the start of 2008. Some go back further. There are approximately 10,000 episodes from dozens of programmes, each with links, images and embedded video. The images we all stored in Matrix and also had to be transferred.

Because Matrix pages are only assembled when they are requested publicly, the most practical option was to download each page and scrape the content. We have been very consistent in the HTML mark-up used on the site, so this would be reasonably successful.

The first task was to get an inventory of URLs for each programme. A manifest file was created for each programme that gave a list of all the unique episode URLs for that programme, and optionally (if it existed) the episode summary from the metadata. I was not able to rely on programme schedules to get this information because some programmes run specials (Morning Report and Checkpoint) and others were cancelled due to Civil Defense events.

In Matrix I used one Asset Listing for each programme to generate a manifest in XML, and wrote a script to read each of these and cache them locally. Once available, a second script requested every URL listed in the manifests and cached these as well.

Caching the files locally allowed much faster test runs, eliminated unnecessary load on the server and also allowed me to quickly make small tweaks to the HTML if required.

The next step was to extract the core content from the page. I used Nokogiri as my tool of choice for this task.

All body content on the site sits within a div with id #cont-pri. After extracting this block from the page unwanted elements were removed from the DOM.

Audio was removed completely, as this is linked to episodes by association, and rendered dynamically by ELF. Other content to be added later (such as promotions of future content) was also removed.

Host information (in a paragraph with the host class) was extracted, and the spelling of some names was corrected. For each programme I ran the script with debugging code in place to list all episodes where the host could NOT be extracted. Once identified I updated the cache files, or altered the import routine to allow for the variation.

During the actual import the audio and relevant presenter were associated with each episode and saved.

Images

The import script also iterated over all images in the content, cached them locally, then uploaded them into ELF, and changed the link in the HTML to point to the new image.

The import routine was designed to run on the same content repeatedly without creating duplicates. This made it simple to re-parse and import the content again if something incorrect was found later.

As it happens we did find a couple of markup issues and ran the importer on our live system, under normal load with people browsing the content.

I have posted the code on github. I make no excuses for the ugliness or levels of hackery. This code is of the get-it-going-fast-run-it-once-and-throw-it-away-while-also-learning-ruby variety. Don't expect too much.

The clean_up_episode_html is of interest as this is where the captured HTML is cleaned up. A series of regular expressions are used to find and replace mark-up generated by the WYSIWYG in Matrix that is not what we'd ideally want.

The HTML is then sent to html tidy and a smaller set of regexs is used to fine tune the code.

This function was fine-tuned by running episodes though it and watching the output for anomalies. Existing regexs were adjusted, or new ones added to clean the output.

Now we have most regular Radio NZ programmes running in ELF: Morning Report, Nine To Noon, Midday Report, Afternoons, Nights, Saturday, This Way Up, Sunday and Arts on Sunday.

Many of Radio NZ Concert's programme are in ELF too: Upbeat, Music Alive, and many others.

Now that most of the hard transfer work is over, I am focussing on building out functionality to support programmes that don't fit existing patterns. Examples are Enzology and New Horizons.

I am also working on improving the administration section - more programmes means more users and this is generating good ideas for improvement.

I'll be posting only fortnightly on this topic from now on, as I have nearly caught up with the work currently being done on ELF.