How to test for audio tag support in your browser

I am writing a small application at the moment that needs audio tag support for Ogg in Firefox 3.1.

The following is the snippet of code I'm using to test this. (I am using the jQuery library).

audio_elements = $('audio');
if('volume' in audio_elements[0] ) {
// processing here
}

This assumes that there is only one audio element on the page (in my case I dynamically insert one). If the volume property exists, I assume that the element can be used to playback Ogg Vorbis files.

There may also need to be a check that the browser actually has the codec installed. (Add a comment if this is the case, and I'll update this post when I work out how).

Update (19 Nov 2008): A better way is to add a check for the Mozilla browser into the test. At the moment, Safari has the audio tag, but only supports media types that work in Quicktime.

if('volume' in audio_elements[0] && $.browser.mozilla)

A new function has been added to the HTML5 draft spec - canPlayType() - that gets around these problems.

Javascript console syntax error on Doctype

I just solved an interesting problem.

There was a Doctype syntax error in the console.

This was caused by an invalid javascript file included on the page - invalid meaning there is no file in the src tag, or the file does not end in .js.

The fix was to remove the CMS code that was inserting an empty javascript tag with no src file.

Tricky to track down, simple to fix.

Converting from CVS or SVN to Git

This post is a collection of notes about moving from CVS or Subversion (SVN) to Git.

Over the last 9 months all my projects have moved to Git. Previously I've used SVN, but found creating and merging branches (which in theory looks like the Right Way to Work) to be pretty painful.

Why Change ?

"If it ain't broke don't fix it" is the most common reason to not change, closely followed by the lost productivity during the change over. Even given these factors, if you closely examine the workflow that older repositories force business into, the benefits of change become obvious.

Imagine that you have 10 developers working on a project in CVS/SVN. A typical workflow is as follows:

1. A new set of features is assigned to 5 of the programmers. The other 5 are working on bug fixes.

2. Each programmer checks out the current head of the trunk and starts work on their bug or feature.

3. Everyone works on code locally until their work is complete, but NO-ONE commits anything as they work, because that would put the head in an unstable state. Work is only committed when it is complete.

4. The first bug fix is complete and is committed.

5. The first feature is committed.

6. Unit tests are run, and all seems well.

7. The second bug fix is committed.

8. The second feature is committed.

9. The unit tests fail horribly and the application appears to be broken in 10 places.

10. The four programmers who made the last 4 commits have a meeting to see what went wrong. As a result the head is frozen while one of them works out how to fix all the problems.

11. While the head is frozen, everyone else continues to code against the checkout they have (now 4 commits ago) while the current batch of problems are sorted out.

12. The code at the head is fixed, and the next feature is committed.

13. This feature causes more unit tests to fail, and re-breaks 3 of the bugs that have just been fixed.

14. The head is frozen again while the current crop of bugs are fixed.

The above is a amalgam of stories I have heard, and appears to be quite typical in many shops using CVS/SVN.

There are many negative things about this workflow:

1. When features and bugs are committed, they can create more bugs if they clash with other features that have already landed on the trunk.

2. Commits can cause complex bugs because of amount of code in a commit is large.

3. All the work that went into a bug or feature is contained in one commit. If a feature took 5 days of work, this is a lot of change for a commit and can make it harder to identify the point that something when wrong.

4. The inability to commit (i.e. save) work in progress tends to reduce experimentation.

There is also the problem of merging. Nearly everyone I know says that it takes a lot of time to plan a merge, and in practice many people avoid branching as a result.

Source code management systems should not create extra work.

The Git Workflow

Contrasting the above, Git allows for simple branching and merging, making it trivial to work on new features and bugs in a temporary branch. It also makes it simpler to manage existing stable, development and maintenance branches.

While on a branch, the programmer can make commits as they work. They can branch from the branch to try an experiment. They can roll-back to any previous state. They can even go back to the trunk (master) and do a quick bug fix, before returning to their current work.

The whole point of source code management is to capture work in a progressive manner at a granularity that is useful for understanding the evolution of a feature, and to help in tracking bugs down. (The bisect feature in Git is great to find the point where code broke).

When the new feature is complete, the programmer can rebase their work off the head of the master branch (the trunk). A rebase takes the current (feature) branch and all its commits and moves it to somewhere else. If you rebase a branch off the main truck of code, it is the same as if you had made the branch off the current head, rather than 20 commits back, which is where you started.

This enables local testing to take place just before the new feature is merged back into the trunk.

Practically speaking, a programmer making a new feature would do the following. I'll include the Git commands required, and I'll assume that the programmer already has a local copy of a remote master repository.

1. Create a new branch

git checkout -b new-feature

(-b creates the named branch)

2. Start work. Create a new function as part of the feature. Commit the function.

git commit -m "This function is to add some stuff to blah"

3. Rebase (the master branch has had 3 commits since work started).

To do this they switch back to master

git checkout master

Then changes from the master repository are pulled and merged locally.

git pull

Then checkout the feature branch

git checkout new-feature

The rebase it.

git rebase master

(There are faster ways using fewer commands to do this, but I have broken it out so you can follow the logic.)

4. The programmer then runs locally the unit test for the module he is working on.

5. The unit test fails. Because the change made in the commit is quite small it takes only a few minutes to find the problem. The bug is fixed and committed.

git commit -m "Fixed bug caused by changes in module Y"

6. The cycle above continues until the feature is done.

7. The programmer then switches back to the master branch

git checkout master

8. And merges in the changes.

git merge new-feature

9. All unit tests are then run locally. The code passes so the changes can be pushed to the main repository.

git push

The main repository now has the new feature, AND any other changes committed by other programmers, and assuming they used the same process, the HEAD is now in a working condition.

The git workflow avoids all of the problems that arise from working in isolation, and the single large commits. It allow much greater flexibility to experiment, and to test changes against the current trunk at any stage.

It also means that the evolution of all features is available in the repository.

How to Change to Git

There are two ways: cold turkey or slowly. If you go cold-turkey it has to be on a new project (easy), or you have to convert your old repository (harder).

The slow way is to use an adaptor like cvsimport or git-svn.

Personally, I would recommend cold-turkey.

SVN Cold Turkey Links

Basic Migration

Migration to a remote server

Project and server migration

CVS Cold-Turkey Links

CVS to Git transition guide

Understanding Git

I'd recommend watching the following videos before starting to use git.

Linus Torvalds on Git

In this video Linus Torvalds explains the rationale behind Git, and why the distributed model works better than other models.

Randal Schwartz on Git

Randal Schwartz explains the inner workings of Git.

Git with Rails

Ryan Bates shows how to use Git with a simple rails project. This is useful to see how easy it is to use.

Installing Git

On GNU/Linux (from source)

On OSX

On Windows

Learning Git

To learn how to use git, the best videos around are on gitcasts. To get started view the first 4 videos in the basic usage section.

There is also a guide for svn deserters.

Tools to help using Git

The best tools are built right in, you just have to know where to find them. My personal favourites are command line auto-completion and showing the branch in the prompt. I have blogged about this previously.

Those of you who are used to CVS/SVN may not see the point of the second of these; once you start using Git you will be making and merging branches all over the place, and the prompt is a great reminder of where you are. The prompt that comes with Git also shows when you are part way through a merge or rebase (i.e. you have unresolved conflicts).

You can use gitk to view the repository (or gitx for OSX).

If there are other resources that readers find useful put them in the comments and I'll add them to this post.

Native Ogg playback with Firefox 3.1

I have added a new Javascript function to the Radio NZ site called Oggulate.

This function parses through pages that have Ogg download links and replaces them with a Play/Pause button. The button plays (and pauses) the Ogg file using Firefox 3.1's built in support for Ogg Vorbis.

The Oggulator can be activated by installing a small bookmarklet:

javascript:oggulate();

Clicking on the bookmarklet runs the function and gives you a nice page full of buttons to play Ogg. You will need one of the latest Firefox 3.1 builds for this to work.

The functionality in the script is rudimentary, and 3.1 is still in development, so don't expect it to be perfect. (I notice there is often a delay before playback starts for the first clip you play on a page, and I have had a few crashes and lock-ups).

This feature of the RNZ site is experimental at the moment, so I'm not able to offer support. It is primarily so people can test Firefox.

If anyone wants to add features or improve the function, I'll upload it (after reviewing the changes) so everyone can access them. webmaster at radionz dot co dot nz.

(NB: As at 2013 this feature is no longer supported.)

Clearing the Cache on Matrix

Here's the problem:

We run a master/slave server pair. Each has a web server and database server. The master is accessed via our intranet, and is where we do all our editing and importing of content. This is replicated to the public slave servers.

The slave server has a Squid reverse proxy running in front of it to cushion the site against large peaks in traffic. These peaks occur when our on-air listeners are invited to go to the site to get some piece of information related to the current programme. The cache time in Matrix (and therefore in Squid) is 20 minutes.

The database is replicated with Slony, while the filesystems is syncronised with a custom tool based on rsync.

If we update content it can take up to 20 minutes for that content to show on the public side of the site. This is a problem when we want to do fast updates, especially for news content.

We've looked at a number of solutions, but none quite do what we wanted.

In a stand-alone (un-replicated) system clearing the cache is simple. There is a trigger in Matrix called Set Cache Expiry, that allows you to expire the Matrix cache early. This works OK on a single server system but not if you have a cluster and use Squid. The main issue in that case is that even though the trigger is syncronous, the clearing is not. If Slony has a lot of work to do, there is still a chance that the expiry date has passed before the asset is actually updated on the slave.

A clearing system needs to be 100% predictable, which led me to devise an alternative solution.

We needed to do three things:

a) Determine when changes made on the Master have been replicated to the Slave.

b) Collect ids of assets that are changed and the pages they appear on.

c) Clear the assets collected in b) when we know a).

This is how we do it.

a) There are two queries that can be run on the Master database to get this information:

psql -U postgres -h server_name db_name -qAtc "SELECT st_last_event FROM _replication.sl_status"

returns a sequence number which represent where the Slony master is currently at.

psql -U postgres -h server_name db_name -qAtc "SELECT st_last_received FROM _replication.sl_status

returns the sequence number where the slave is up to.

If you grab the master sequence number after a content change (a database query), you can tell when that change has reached the slave when it's sequence number is the same or greater.

b) We have a script that imports news items to matrix. One of the attributes in the imported data is a list of asset ids that are affected by the import action. We know in advance what asset lists and pages the content will show on.

When the script runs it collects these for each imported asset, and compiles a list of asset ids (with no duplicates).

c) This is how it is bolted together.

After some assets have been imported, the import script calls a second script which adds the items to a queue:

system( 'perl add_to_cache_queue.pl --assets="' . $asset_list . '"' );

This script is short, so I'll reproduce it here.

#!/usr/local/bin/perl

# This script is used to add items to the DirQueue on the current machine
# it is for testing purposes

use strict;
use Getopt::Long;
use DirQueue;

use lib ".";

my $assets_to_clear = '';

GetOptions( "assets=s" => \$assets_to_clear );

if( $assets_to_clear eq '' ){
exit(1);
}

my $command = 'psql -U postgres -h host db -qAtc "SELECT st_last_event FROM _replication.sl_status"';

my $last_event = `$command`;

$last_event =~ s/\n//;

print "Last event: $last_event\n";

# a queue to add items to. Locks can last for 2 minutes

my $string = " this is a test string add to the file at " . time ."\n";

my $dq = DirQueue->new({ dir => "matrix-cache-queue",data_file_mode => 0777, active_file_lifetime => 120 });

if( $dq->enqueue_string ($assets_to_clear, { 'id' => $last_event, 'time' => localtime(time)} ) ){
exit 0;
}

print "could not queue file";
exit 1;

A second script runs on the machine as a worker process, watching the queue. This uses the loop code I outlined in my last post.

The queue itself is Perl's IPC::DirQueue, a very cool module for managing a filesystem-based queue.

When an item is found on the queue it checks the Slony sequence number that was saved with the data. If the number has passed on the slave, then yet another script is run, but this time on the slave (public) server.

php ./matrixFlushCache.php --site="/path/to/site/radionz" --assets="comma_sep_list"

This last script resolves the asset id numbers in Matrix to a list of file system cache buckets and URLs. The cache buckets are removed, and the URLs are also cleared from the Squid cache. The cache is them primed with the new page. The script was written by Colin Macdonald.

This is what the output looks like:

** lock gained Sat Aug 30 20:03:01 2008 running jobs: * * * * ?
Slony slave is at 485328, Got a job at: 485327
data:200,1585920,1584495,764616,etc
Clearing cache for #200
Unlinking cache/1313/e8f12c0d7b5889d748872bdad215c0cf
Unlinking cache/1113/aaa1666c26af81a0b044ab2fecb950ae
Deleting DB records (2 reported, 2 deleted)
.
snip a bunch of the same but for different assets.
.
Refreshing urls:
http://www.radionz.co.nz/ ... 200
http://www.radionz.co.nz/home ... 200
http://www.radionz.co.nz/news/business ... 200
http://www.radionz.co.nz/news/business/ ... Skipped
snip a bunch of URLs
- lock released

The script is looping and checking every three seconds for a new job (queued asset ids to clear). The * means it checked for a queued job and none was found. The ? means that a job was found but that the slave had a lower sequence number than the one stored with the job.

The top 5 stories (the ones on the home page) are also cleared and refreshed.

The Flush cache has a URL filter so you can exclude certain URLs from being flushed - an example is the query ridden script kiddie hacks that people try to run against sites. There is no point in re-caching those.

Another is URLs ending in /. In our case this mostly means the someone has deleted the story off the end of the URLs to see what they get, so there is no reason to refresh these either.

A feature that I'm working on will clear just the Matrix and Squid caches for the non-front page stories. These all have a 2 minute expiry time and if we expire all the caches the end user will re-prime the cache. There is no performance hit in doing this as the browser and squid come back for these pages every two minutes anyway.

The system I have just outlined allows us to remotely add and update items in Matrix and for those changes to appear on the site within 5 minutes. I hope someone finds this useful.