During the course of the project to replace MySource Matrix with a bespoke solution (ELF) based on Rails I've experienced doubt, frustration and regret. Doubt that we'd ever finish the project, frustration at the lack of progress at times, and regrets about the past.
Doubt
Twice, so far, the complexity of ELF development has peaked, and with it doubt has set in. I really did wonder if we were going in the right direction, and if we'd be able to implement all of the functionality we needed.
In theory, the main advantage of the agile development process is building on small wins and the ability to correct mistake early. Building on the success of previous iterations helps reinforce that you are on track. In practice this is also true, but at times I found my self comparing the two systems, even months after we'd decided to make the change and had devoted weeks to coding the new system.
In the early stages of the project ELF couldn't do that much, whereas Matrix was still doing everything, so the comparison was not very favourable. Things that were quite simple to build in Matrix were requiring quite a lot of thought to implement in ELF.
Looking at why this was, Matrix encapsulates certain high-level patterns in the form of assets. These assets can be bolted together to create complex public-facing pages. In designing ELF we had to look for lower-level patterns, and this took time and effort. Finding the right patterns (and replacing out-dated patterns with better ones) is a constant process.
Once I realised this - that the design of ELF was going to evolve and improve, and that we had complete control of our destiny - the doubt dissipated. We can have the system any way we want it. If a pattern is wrong, we can change it. If some code is slow, we can refactor it. None of these options were available to us with Matrix. (I should note that this is about trade-offs - complete control of the software stack and all that entails versus an out-of-the-box plug-and-play but not quite right fit).
The other area where doubt set in was the migration of content. I started looking at the whole site - tens of thousands of pages, images and audio items - an enormous task. To get over this, I broke the content down into small pieces.
An example of this was image migration. An image linked from a page in Matrix has to be moved over to ELF and relinked in the HTML. I broke this down as follows, getting each step right before moving on:
- Fetch an image from a URL and cache it
- Add this image to ELF returning the new URL
- Get the HTML for a page and cache it
- Parse the HTML looking for images.
- Fetch those images, get the new ELF URL and update the HTML
- Save the HTML to ELF
Frustration
Our site is big. Really big. You just won't believe how vastly hugely mind-bogglingly big it is.* We don't have the resources to do a big-bang migration to the new system - as you've noticed from the rest of the series we are doing it section by section (and later on) programme by programme.
At times progress has been slow as other projects related to on-air content have to take priority, and we are also working on a new design.
Having to work in two systems is a pain - ELF is much faster to update and simpler to use than Matrix because it has been optimised for our precise use-case. The key for me has been to look forward, not back. Each week more content is run out of ELF and with additional people using it I get many suggestions for improvements. We really are in control of our destiny!
Also, Rails continues to develop and add functionality that we can use to enhance the site and streamline the development process. The built-in testing framework ensures code is robust and that minor changes don't break things. The processes to import content have never been as stable or as well tested.
Regrets
Hindsight is wonderful thing. In the first draft of this post I listed some of the things I should have done, issues I should have caught earlier and so on. But that was then, this is now. What is done, is done.
A couple of year ago I was involved in a startup that did not succeed. Fact: start-ups fail.
At Webstock that year one of the speakers said they loved hiring people who'd been involved in a startup, even if it had failed because those people had done something (rather than nothing) and had generally learnt a lot from the experience.
Applying that to this case what have I learnt?
- An enormous amount about content caching
- Linux and basic database administration
- How to highly optimise our markup and CSS to reduce server load (and speed up page delivery).
- Lots more!
Summary
Today as I write this I am looking forward to the day that ELF is running our whole site. In the last week I moved 8 more programmes over to ELF. These were One In Five, Spectrum, Te Ahi Kaa, Saturday, Sunday, Ideas, Insight and Mediawatch - between them 1000 pages of content, 250 images, 15 image galleries and about 6,000 pieces of audio. There were no issues with the migration and the new pages were made live in the middle of the day under normal traffic loads.
Right away we could see the huge improvement in page responsiveness, and the other members of the web team could use the much faster administration section of the site for those programme. That was a real morale booster.
Next time I'll get back to covering the migration of content.
* with apologies to Douglas Adams
No comments:
Post a Comment