Tuesday, June 28, 2011

Rebuilding Radio NZ - Part 11: Editing Episodes

I am jumping ahead to compare our new episode editing interface with the old one, leaving the implementation and migration of episodes for next time.

This is the top section of the programme episode editor in ELF, showing Nine To Noon for 16 June 2010:

It shows a link to the host, the current status of the page and edit/trash buttons. Audio for the episode is displayed on the page, and can be edited directly from there. In edit-mode text content is entered directly into a WYSIWYG, and the host can be changed.

We are using the CK Editor in ELF, and this is very good at removing HTML cruft from content pasted in from Word and Outlook.

In addition to the native clean-up, I have added an additional formatting function for the body. This takes the body content, sending it back to the server to be pre-formatted by ELF's built in parser/formatter. The parser takes the input HTML and returns content formatted in a standard way (bold for lines that start with time). A future version will add links when RNZ programmes are mentioned in the heading.

This may seem a pretty minor feature, but it saves a huge amount of time on a core content task. That is a primary design concern in ELF - eliminating mundane tasks so that users of the system can work on value-added tasks.

Images can be added via a button in the WYSIWYG, and uploaded directly via an image browser. The image browser immediately shows only those images for the current programme.

I should note that everything I have shown is a work-in-progress. The body parsing function was updated last week to just parse the selected content to allow new content to be added to existing content and formatted in place.

ELF generates all progamme information such as the date, host and broadcast times on the fly for every page.

This process differs a lot from the interface in Matrix. Navigation to an episode is via the tree, and the episode is stored in a standard page (shown here without the tree):

The name of the programme, the host for the day and times of broadcast are text content. The audio content is inserted into the page only when it is publicly requested, and cannot be seen in this view (it can be previewed, but not edited from there). The status is on another screen, accessed via a drop-down menu or right clicking on the asset tree and so is the field for a summary of the programme.

Pasting into the WYSIWYG is hit and miss (in our version) often requiring manual editing even after the built in cleaner is applied. I suspect we are stricter than most in what we'll except in our markup.

Image loading takes place in another place tree context (or via a simple image uploaded interface). Once images are loaded they can be added via a button in the WYSIWYG. This launches the image browser which has a tree for navigation.

General versus Custom

Looking at the two approaches, it is clear that in our case an interface that closely matches our workflow and excludes superfluous options results in a better user experience. This was one of the key drivers for change. The cost of the bespoke approach is that the system has to be designed, coded, maintained and supported.

The Matrix approach provides a large number of general tools, allowing sites to be built without any code being written at all. The functionality to build most things is built right in. The downside is that in the administration interface is it not possible to hide unused functionality, and it may not be easy to group related content together in a way that matches business processes. The new version of Matrix (the Mini) does a very good job of fixing both these issues, and the integrated context sensitive help is very impressive.


The ELF interface I have shown is the product of dozens of iterations, many of these based on feedback from colleagues. The first round of feedback came from the web-team - they were the first to use the interface for day-to-day work. After the first round of improvements we've started training producers on the system. Training sessions have ranged from 5 to 30 minutes, and during these we've taken notes on what didn't work so we can improve the system further. 

An example is ELF's episode navigation. It is fine for daily programmes where the navigator has five out of seven days linked. It does not work so well for weekly programmes where one in seven links is active, or yearly programmes where one in 365 links are active. In these case there is too much scrolling, and in the case of yearly programmes it is hard to find the previous episodes. I don't have a solution yet.

Some other rough edges

While the interface is fine for once-a-day use by producers, there are some problems that only became evident when using the system all the time and repeating the same actions again and again in a short time frame.

While setting up the documentaries section in ELF I noticed that the workflow was not right - for example the layout of settings on the programme setup page was not intuitive. Not all programme types require all options, and the page needs to have these shown and hidden dynamically in response to changes to the programme type setting.

After using the interface many times, there overhead of having to hunt for the right thing to click or set starts to add up. The benefit of a bespoke system is that these things can be changed

In the next installment I'll cover the huge task of importing thousands of past programme episodes along with audio links, images and presenter information. Stay tuned!

Saturday, June 18, 2011

Rebuilding Radio NZ - Part 10: Going treeless and Modules

One of the biggest time wasters (for us) in Matrix has been navigating the admin section of the site via a tree. Trees can be useful of many tasks, and for seeing the hierarchy of assets (and URLs). But when the number of items in the tree increases above a certain size it becomes harder to get to the one you want.

I should note that Matrix does have a 'simple edit interface' - a way to directly access the editing screen for a particular page - but we've never used it because the site was (and is) growing too fast to set these up for sections and pages. Work is done by site admins (site-wide) or producers (one programme), so the fit was not quite right.

Take Nine To Noon as an example. This programme has 1200 child pages - one for each day’s episode (see image). To get to the current day’s page you have to open the Site node, scroll down, open the station node (National), scroll down and open the programme node (Nine To Noon), wait for the the children to load and display, then scroll down to the bottom of the list. Whew! You can use key-strokes to do all this, but the loading and scrolling time is still high. Moving between programmes is even more unwieldy.

Another example is the audio folders. Nine To Noon currently has over 7,000 audio items. These were once stored in a single folder. When we got to about 3,000 items we had to move everything into dated sub-folders - year, then month, then day. A script was written (now part of the Matrix core command line utils) to move existing items into dated subfolders.

This structure made it simpler to get to items based on their broadcast date, but much slower to move between assets. It also added unnecessary URL segments to the audio path.

We are adding and editing a lot of content every day. Moving around a tree has a very high operational overhead given the large amount of existing content.

In the early stages of the project Nigel (from AbleTech) and I debated using a tree for ELF. Nigel was dead against it, I was less so provided it could be designed well. I agreed to go treeless for the admin interface and to-date one has not been needed. Nice one Nigel!

The biggest problem with abandoning trees though, is the question of what to replace them with. The major navigation pain-point for us is navigating content by broadcast date.

Most developers' initial answer to the date navigation question looks like this:

There is nothing inherently wrong with these date pickers - they are used in a few places in ELF's admin section. Personally I find them too busy, they all work slightly differently, they require a lot of markup and solid engineering to be accessible, and they make you think. Don’t make me think. The solution we came up with involves a sliding date range selector, powered by jQuery and ajax:

Clicking the left or right arrows takes you forward or back. Valid content links are passed into the widget via an ajax call. The dark red button is today, beige buttons are days with content, and the blank buttons have none.

This widget is used right through the admin section of ELF to move between programme episodes, schedule events, and highlights, and is proving both fast and intuitive.


As the number of asset type increased (Audio, Episodes, Highlights, Schedules Events) we ended up with some duplicated code in our models. These were often slightly different implementations of scopes, coded at different times by different people, all intended to do the same thing, and all tested in slightly different ways. The answer to this problem was to extract the shared functionality into modules.

The first of these modules covered trash. Every trashable asset has a field ‘trashed’, and the module provided consistent methods for trashing and restoring items, and a scope to skip these items in queries.

The second module was for status - live (public), and under construction (hidden). This wraps the published_at field and control visibility.

The third module contained scopes for selecting items based on broadcast date and time - latest, now, between, and for (a specific date).

Having the code in modules simplifies testing and fixed some bugs caused by the subtle differences in the original many implementations. They key lesson is to look for patterns and common code and extract them into modules.

I put the modules into lib, however DHH published a Gist a week ago suggesting their own folder. And if you are not using ActiveSupport::Concern for this sort of thing, you should be. Here is an explanation.

Rails 3.1

Over the last few weeks I have started work on moving the app to Rails 3.1. The feature that is initially of most interest to us is the asset pipeline. DHH first mentioned this in his 2010 Rails Conf Keynote, and this talk was the inspiration for our CSS Views Gem, coded by Koz of Southgate Labs and released on github.

The asset pipeline is a no-brainer. It allows better organisation of CSS and Javascript, and packaging and compression/minification in production. CSS and Javascript are now first-class citizens in the framework  and can be mixed in with Erb, SASS, CoffeeScript or the language parser of your choice.

Having a fast site is important to Radio NZ - I have spent many hours improving markup and streamlining how content was packaged and served - all to great effect. Over the last few years we've reduced page size by 30% and halved client-side rendering time. Doing this in Matrix has been quite hard due to the way CSS and JS content handling is abstracted. You have to get under the hood and hard-code stuff in the Matrix templates to get the highest benefits.

Rails on the other hand is moving towards being 'fast by default'. A selection of sensible and safe best-practices that will work for most use-cases are baked into the framework and turned on by default.

Next time I'll look in more detail at the editing interfaces in the ELF admin section, and compare them with the work-flow in Matrix.

Friday, June 17, 2011

Upgrading to Formtastic 2.0

I have been using Formtastic 1.2 for the Radio NZ site rebuild. I have a number of custom input types, customised via extending SemanticFormBuilder and adding my own methods. Justin outlined the changes a few months ago.

Formtastic 2.0 RC just got released, and Justin points out:
Folks who subclassed SemanticFormBuilder and created their own custom inputs as methods will be in for some pain.

OK, I have been through the pain, and it wasn't so bad. Here is what I did:

1. Changed all instances of Formtastic::SemanticFormBuilder in the formtastic initializer file to Formtastic::FormBuilder.

2. Remove the custom builder declaration in the config file

3. Moved my old custom methods into a module.

(I added include FormtasticExtensions to the top of the initializer file.)

4. Converted my methods to classes.

I have posted a gist of the converted module with the old methods below the new classes.

You should also change the setting of all_fields_required_by_default to false if you did not have it set before, otherwise you'll find that forms that once saved will have HTML5 warnings for all fields. The other gotcha is that validates_length_of without validates_presence_of in the model makes the field required. This might not be what you expect!

My impression of the update is good. This custom code is a lot simpler, and less hackery is required to get what you want.

I found one issue I am still working on fixing - new forms that use the CK Editor for text areas won't save. This happens if the field is required. CK Editor does not write any content to the text area, so the form won't save. A work-around is to mark the field as :required => false and rely on the server-side validation to display the error to the user after trying to save.

Hopefully this helps others with their transition to 2.0.

Saturday, June 11, 2011

Rebuilding Radio NZ - Part 9: Highlights

Each week Radio NZ publishes highlights of upcoming programmes for print media, and these items are also used on the website. On the website these highlights are augmented by content from programmes with production cycles that make it impossible to meet print deadlines.

Highlights appear on the Radio NZ Home page, National home page, and Concert home page. Highlights are also display on programmes pages - for example at the bottom of the Insight page. Their use on programme pages meant this section had to be migrated before I could start on programmes.

Major refactoring

Highlights and Schedules were based off the same model (schedule_events), with a few fields being for the exclusive use of one or the other. Schedules had been running for a few months, but when I came to actually use highlights a some extra features were needed. Because the needs of the two were diverging, I decided to split them into separate models.

This made the code simpler to read and avoided having to hide different fields in the edit screens. The other major refactor was the removal of a base model - content - as an association. The content table held common fields such as body, broadcast time, and status.

It is my view that this approach - of using a single table to hold data that is shared between other assets - is almost always wrong. It makes the code more complex and harder to follow. There is an extra join for every database query, and this make indexing and optimisation more complex. Yes, there is duplication between models - many have a body field - but the benefit is clarity and ease of maintenance. In our case we wanted to keep things simple and avoid overhead.

To avoid code duplication, the functionality built on these fields is extracted into Modules. I'll cover that in a future post.


There was no need to migrate historical content; only the content for the next few weeks is available on the site. This was extracted via the same XML technique mentioned previously, and imported into ELF. For a short period we generated XML content feeds in ELF for use on some Matrix-generated pages.


In Matrix highlights are displayed based on their location in the tree. In our case we had folders inside the main highlight folder (right). Each of these programme highlights folders had to be linked to each programme folder. Linking is basically a reference to the same folder.

Each page that requires certain highlights has its own asset listing. 
The disadvantage of all these asset listings is that they each had their own cache expiry based on the most recent build time. It was quite common for a highlight to be added and appear on some pages and not on others for up to 20 minutes (the cache time).

You can have all assets of one type in one place, but this almost never happens in practice. 

In ELF all highlights are (obviously) stored in the highlights table. The relationship to each programme is an explicit association:

  belongs_to :programme

This allows us to display highlights by programme or by station (every programme has a station). This is much simpler than a tree based system as you can always tell how many highlights a programme has.

Instead of multiple asset listings for duplicated HTML, there is one html template (a partial) for the display of all highlights in programme or station context.

Changes to highlights in ELF update everywhere as soon as they are saved.

Highlight Administration

In Matrix, adding highlights was a multistep process.
  1. Go to the correct folder
  2. Create a News Item
  3. Enter a summary and body on the details screen
  4. Change the created date and time to match the broadcast time of the asset  
  5. Update the metadata to provide a link for the RNZ email newsletter
  6. Make the item live
In step 4 we are faking the broadcast time in the created field. Everywhere that highlights appear, they are sorted by created date and time, and only those after 'today' are displayed.

Individual programmes could enter their own highlights by adding News Items to a folder in their part of the asset tree (as in this example from Saturday Morning).

ELF Administration

In ELF there are two contexts for adding highlights. The administrator context allow highlights to be entered for any programme, while the programme context allows individual producers to add highlights  for just their programme. This is the edit screen:

Everything is on one page, and images can be uploaded directly without having to go to another screen. There is also a widget to move between days (it is not a tree), and I will reveal this in a future post.

The speed of adding highlights is much faster in ELF. I taught a producer to add their own highlights last Friday; it took less than 30 seconds to create and add a highlight.

These are the benefits of a well designed bespoke system - simpler maintenance of content, faster updating, and less confusion for users.

Next time I'll cover going tree-less, and extrating functionality to Modules.