Vert Pixels – app design and more

So, this blog article is admittedly a little late, but we’ve been kept busy. I’ve finally not got the time to sit down and write a bit.

Back in the middle of October 2010 we started work on the iTunes LP for Deadmau5’s new album 4×4=12. The most important and challenging part of the brief was that the label wanted a visualiser that would respond to the audio, and be in keeping with the Deadmau5 image.

By this time we’d written several different visualisers, making use of various technologies. Since viz development takes time and money, the most common option chosen by our customers is usually a simple slideshow. We do like to stretch ourselves though, so these often get augmented with additional bits of fun. For example, whilst Kylie’s “Aphrodite” has a slideshow viz it is framed by animated waves above and below the photos, and track titles that get animated onto and off the screen with a zooming effect when the track changes. The LP for Michael Franti & Spearhead’s “The Sound of Sunshine” has quotes from Michael Franti appearing to accompany the slideshow. Gatecrasher Anthems’s slideshow uses a fancy sharding effect to transition between photos. Iron Maiden’s “The Final Frontier” features four different visualisers, one that shows titles over a spinning starfield, the second is slideshow of some of the artwork created for the album showing their mascot Eddie and other aliens, third a colour-shifting photo of the band, and finally a 3D ‘skull tunnel’ viz inspired by the 80’s arcade game Tempest. (For good measure, the front page of Iron Maiden also features blobs of blood floating around space.) On a related vein, we have done timed lyrics displays for several LPs for singles, like Roll Deep’s “Good Times” and Professor Green & Lily Allen’s “Just Be Good To Green”.

There’s a few visualisers we’ve developed that for various reasons have not seen the light of day too. We had a viz that showed short fragments of lyrics showing up timed to the music, fading in and out as the songs are sung. Another that didn’t make it out was a fish-tank viz that featured several fish swimming about the screen, bubbles floating up, and lyrics that floated up the screen in time to the music.

Technically there’s a lot of different technologies going on in that lot. Many of them make use of dynamically generated HTML. We make extensive use of CSS3 animations, transitions and transformations, including 3D transforms – indeed the Iron Maiden ‘skull tunnel’ uses nothing but those techniques. The waves in Kylie’s LP are produced by JavaScript code using HTML5 canvas. That code monitors how long the system takes to draw those waves to ensure the wave animation is as high resolution and smooth as they can be – run that LP on a low powered PC like a netbook and the waves will be smoothly animated in low resolution, but run it on the latest and greatest and you’ll see full 720p resolution waves running at 60 frames per second.

Whilst we had done plenty of things timed to happen at specific times in tracks one thing our LPs hadn’t featured before starting Deadmau5 was a viz that directly responded to audio. We’d produced a few experiments for our own benefit that drew spectrum graph displays but they weren’t ready for prime-time; they looked a bit naff and didn’t really work very well. But now we had some time, some budget, so it was time to brush off that code, and get engrossed in some serious research and development. So I got to work.

I soon discovered a few tricks for dealing with the waveform spectrum data that one can extract from iTunes. The key to processing the data to make it look better is to group together values into separate channels. iTunes provides 256 useful level values over a linear spread of frequencies, however that’s not how we perceive audio. We hear sounds instead on an exponential scale; a middle C tone has a frequency of approximately 261.5Hz, C an octave higher is twice that at 523Hz, and C an octave higher still is 1046Hz. To produce a spectrum graph with 6 bars/channels for example, the 255 data values from iTunes’ data need to be grouped into blocks of 1, 2, 6, 18, 55, and 174 values. Armed with that piece of knowledge I set to work producing some prototypes.

Before long I had some simplistic spectrum bar graph displays running that, whilst quite crudely styled, gave a response that looked right. A steady stream of prototypes ensued, and little later they were looking like the LED displays popularised on stereos in the 80s, complete with decaying peaks.

Now prototypes are useful, but they’re not an end-product. We needed to come up with something that would look “Deadmau5”. If you look up Deadmau5 on YouTube you’ll find several live performance videos of him. There’s a distinct Deadmau5 visual style – he stands (wearing a large Deadmau5 head) behind a tilted cube-shaped DJ booth, where the faces of the cube are display screens, and there’s generally also a large display screen behind him. Various different animations and effects get displayed on these screens. These videos gave us plenty of ideas, something to aim towards.

Since the cube-shaped booth is an important part of the Deadmau5 stage performance experience we had to include that into our viz. Technically the cube is three HTML5 canvases rotated in 3D space using 3D CSS transforms to form the three visible faces of the cube. We had two parts of the puzzle – we knew how to do viz things that responded to audio, and we could do a cube display.

What was clear at this point is that in order to create an experience that looked like a Deadmau5 stage performance we were going to need a sophisticated viz engine. It was going to need to be timeline driven, and for that we could leverage off our previous experience with timed lyrics displays and timed commentary. The engine was going to need to support rendering to multiple different canvases, enable and disable different displays, and support several different types of animation with different display options, including of course spectrum bar displays.

I’ll go into some more specifics in a future blog post, but the high level overview is this:
The viz engine for Deadmau5 features a timeline management system coupled with a programmable rendering pipeline system.

The timeline system allows for actions within the engine to be called specified certain times during track playback. Each track has a separate timeline defined specifically for that track, and the timeline being followed will change when the track changes.

Rendering pipelines are defined at run-time. A pipeline comprises of multiple stages, each of which specifies a renderer along with a destination canvas and optionally also a source canvas. Additionally pipelines can be defined to call actions before and/or after the rendering stages are completed. Pipelines are usually started and stopped by actions called by the timeline system.

The Deadmau5 viz engine includes eleven different renderers. The simplest of these is the ‘clear’ renderer, which will clear a canvas, or fill it entirely with a specified colour. The most complex renderer is ‘vu’, which will generate various different VU displays. There are also a couple of effects-type renderers in the form of ‘dotty’ and ‘pixelate’, which allow for various different pixelation effects.

As well as pipelines, the engine provides a ‘settings’ system allowing for settings objects to be created that contain values that will persist and can be re-used between different rendering pipelines or actions. As well as adjusting and clearing settings, there are actions in the engine to increment values, or randomize them.

Finally there is a ‘transition’ system in the engine. This allows for settings values to be changed over a specified period of time as playback progresses. As well as transitioning numbers from one value to another, the transition system also supports transitioning colours, and includes a callback mechanism to allow a set of actions to be called when a transition has completed.

The engine design was modular to allow for things to be combined in most ways we can think of. Even with just 11 renderers there’s a very large number of possible displays the Deadmau5 viz engine can produce. We by no means exhausted those possibilities in Deadmau5…

In terms of raw statistics, the viz engine in the Deadmau5 LP weighs in at approximately 1700 lines of code, plus a further 2800 lines of data definitions for pipelines, settings, and timelines.

In my next blog post I’ll write in a bit more detail about the engine’s pipeline system and some of the renderers included in the Deadmau5 viz.

Development of the engine also hasn’t stopped with the Deadmau5 LP. The viz in the LP for Hillsong United’s “Aftermath” album makes use of a later version of same engine, albeit with less renderers (but including one new one) and a more sophisticated transition system. I’ll also be writing on how the engine has subsequently developed, and giving some insights into where it’s going.