Annotate music for your listeners using audio tag and JavaScript

[Frank Hecker]

I don't usually post here because I'm not a composer or a musician, just a (relatively uneducated) listener. However I'll beg your indulgence today, as I thought at least some of you may be interested in this topic and I do know something about it (I work for Mozilla).

As you may know, many modern browsers, including the latest versions of Firefox (3.5) and Safari (3.1 and later), support new HTML5 <audio> and <video> tags that allow easy embedding of audio or video content in a web page without the need to use Flash or other plug-ins. What you may not know is that you can use JavaScript in combination with other web technologies (CSS, XML, etc.) to build additional functionality around the playing of audio or video content.

As part of a "Firefox 3.5 for web developers" series of blog posts we've just published a post on exploring music with the audio tag highlighting a demo produced by Samuel Goldszmidt of IRCAM that shows how you can add text descriptions for the benefit of someone listening to a piece of music (in this case an extract from Florence Baschet's StreicherKreis).

In the demo the JavaScript code is tracking the current time within the piece to show an indicator of which segment is being played; by mousing over the segment the listener can get a text description, and by clicking on the segment have the segment be replayed from the beginning. Not anything you couldn't also do in Flash, but the point is that you don't need Flash, just JavaScript.

In this case the music is in Ogg Vorbis format (not MP3) because that's what Firefox supports natively; we're trying to encourage adoption of open audio and video formats on the web. You can play the demo in Safari as well (I tried Safari 4.0 on OS X), if you install the XiphQT component allowing playing of Ogg Vorbis in QuickTime-based applications.

If you look at the actual source for the demo page you'll see that at the moment doing this sort of thing is not for the faint of heart, as it definitely requires some JavaScript knowledge. I hope that as more people make use of the and tags developers will create toolkits and JavaScript libraries that will make it much easier for relative web novices to post their own music with text annotations and additional material to supplement the listening experience.

[adam kondor]

A useful tool and an interesting idea!

[Jack Pickett]

Thanks Frank - I run what I hope is cleverer that most amateur music websites using php and mysql databases to allow users to browse music by genre and leave comments on a forum that I built last year. In truth, not many people use the site for this but I enjoyed the process of putting it together and have been able to use the system on some other more successful sites. Overall I have enjoyed the process offering me slightly more solid ground to operate on than the creative life of composition and, personally, this balance is a positive thing. My website uses xml made dynamic by php to deliver a playlist to a flash player. I am currently working on a way to use javascript to display a message if there is any problem with the delivery of the flash player but I am finding javascript a tricky code to get my head round. Sadly after html, css, php, mysql and actionscript 2 & 3 my tiny musician brain is addled!

Nevertheless this solution does seem very useful and probably something I will work towards in the medium term. Thanks for bringing this to my attention - on the whole musicians seem to struggle with this side of the presentation of their work!

[Richard Cooke]

Hi Frank

It is so good to have a developer on board, especially one from my favourite browser.

One problem with new web technologies is that, as soon as amateurs like me have acquired the necessary skills, the technology tends to sink without trace. Beatnik, Zel and the (audio-tagged) Adobe SVG plug-in were works of pure genius but where are they now?

I like the look of the Ircam demo but haven't listened because I uninstalled Firefox v3 months ago to return to v2. That's because v3 rejects the Noteworthy Composer plug-in, and presumably - correct me if I'm wrong - Sibelius and Finale. As is often the case, it's two steps forward, one step back.

What would be really, really useful would be a Firefox audio-tagged implementation of SVG, which would mean composers could display not just programme-notes but musical notes as well. There are examples on my website - for example, http://rcooke.free.fr/iac/vml/fzk.svgz - if you have the time and inclination to download a rival browser and plug-in!

I'm sure, Frank, you know more than the rest of us put together, so please continue to put forward new ideas.

[Richard Cooke]

Hi Jack - If you can write actionscript, you should find javascript a piece of cake! For me, it has always been the most useful of the various languages.

[Frank Hecker]

Richard: Actually I'm not a developer (at least not professionally), I just know a lot of them. With regard to the transience of web technologies, the video and audio tags in Firefox 3.5, along with the means to control them with JavaScript, are not Firefox-specific but rather are part of the HTML5 standard being developed by the World Wide Consortium and being supported by Mozilla (Firefox), Apple (Safari), Opera, and (in future) Microsoft (IE). So I think you are pretty much guaranteed that these techniques are going to be almost universally supported in future browsers, and your personal investment in learning them will not be made obsolete.

The only major source of uncertainty is around the audio codecs that will be supported natively by browsers. For the foreseeable future the unfortunately reality is that Safari will support MP3 natively but not Ogg Theora, and Firefox and Opera will support Ogg but not MP3. (Google Chrome will support both.)

Regarding SVG, my understanding is that SVG is a complex specification and unlikely to be fully implemented natively in any browser; thus you may be dependent on third-party plugins, with all the attendant issues of browser compatibility, etc. (as you're well aware). I suspect that a more "future-proof" approach will be to instead use the new HTML5 Canvas feature, which provides native browser support for simple 2D drawing. Canvas is already well-supported by Firefox, Safari, and Opera (with IE the laggard, as usual), and I'm sure there will be lots of third-party JavaScript libraries to make drawing with Canvas dead easy.

[Frank Hecker]

Jack: Richard is correct, if you know ActionScript then JavaScript is a very similar language, at least in terms of the language syntax. However the underlying libraries are very different; for example with JavaScript to do dynamic effects you need to deal with manipulation of the HTML Document Object Model (DOM), to do real-time interaction with a server to need to do XMLHttpRequest call, and so on. I'm not a Flash expert, but I think the relevant concepts and underlying techniques are quite different with Flash.

In the long-run I think there are only going to be two viable choices for people doing cool stuff in browsers: use Flash, or use the HTML5 family of technologies (HTML5, CSS, JavaScript, Canvas, XMLHttpRequest, JSON, video and audio tags, etc.). Only these two are going to have near-universal browser support. Professional web developers can and will have to spend the time to be fluent in both. Amateur web developers should just pick one or the other. I think in the short run (next year or two) Flash is a better choice for such developers, because it's an integrated solution that pretty much works the same way in all browsers. However in the longer term (post-2010) I think the HTML5 technologies will match and surpass Flash, and may also open up new opportunities for use on devices not supporting Flash (e.g., iPhones and iPod Touch).

[Jack Pickett]

Frank Hecker said:

.....for use on devices not supporting Flash (e.g., iPhones and iPod Touch).

This is precisely why I will be looking to make the move.

[Richard Cooke]

Hi All: I agree, and Frank's option 2 means I shall be taking a good look at Canvas.

To be honest, I have political problems with Flash, as the price means that people in the developing world are excluded. With JavaScript, it's easy to run a database - which is pretty essential if you often add new music.

Jack's site shows that good design can be achieved with or without Flash and I suspect he's a lot better at JavaScript than he admits!

[J.C. Combs]

Hi Frank,

I just wanted to say, thank god for Firefox! :)

[Jeff Harrington]

As a professional Flash and web developer I have to step in and warn anybody thinking of doing this without Flash that that is insane. It'll work for a while on a few browsers, while in Flash it'll work - for a long while. I've stopped recommending jQuery and other JS libraries to clients because Flash is just so much easier to maintain and I'm sick of tracking down IE CSS issues when the DOM explodes. I'm lazy. Doing cool Web 2.0 stuff kept me employed for a decade, but if I had to do any of it over again, I'd use Flash as much as possible. And sure, there's the Google libs and the Yahoo libs, but frankly it's a moving target and with Flash it just works.

As an aside, I would never follow or attempt to develop on top of anything an academic institution does in a browser. Native compilers, sure that's different. They just never seem to be able to accept that web users don't care that it works on browser version X. Your users will leave and never visit your site again because 'it doesn't work.' Academic institutions like IRCAM could care less. Their users have to use whatever browser they tell them to use. That's the beauty of being an academic institution. :)

There is a Flash AS3 compiler for free:

Flash MXML Compiler

I've used it on a number of projects and it is fast and outstanding. I eventually got my client to buy me Flash CS4 because I told them they were wasting time without my having a good debugger.

That's just my perspective. Sounds like an interesting project, but ouch, making complex systems browser dependant is a nightmare that I'd never want to subject anybody to. Plus with Flash you have the builtin MP3 player and a good timer that's not going to freak when another tab does a reload (remember you're not in just a window these days, you're in a complex set of windows, each with their own functioning OS). And you've got great video with Flash and now since Flash 10 3D!

Also forget SVG. That's another hoax that'll never happen. It's like VRML, and the other SGML languages. Give academics a reason to argue. We looked at SVG for years and finally just decided it was great, but Flash just works and everybody has it, so why bother.

[Frank Hecker]

Jeff: I agree that Flash is the best approach for now. HTML5 at this point is a bleeding edge technology, and my point was to illustrate what's becoming possible with open web technologies, not to urge that people abandon Flash wholesale.

What happens in the long run is an interesting question. I'm of course biased given my work for Mozilla, but I suspect that there's going to be a *lot* of innovation around HTML5, Canvas, and other web technologies, and between advances in those technologies and the gradual irrelevance of older browsers (e.g., IE6) I think Flash does face a serious long-term threat to its dominance. The key questions include how fast Microsoft implements HTML5 et.al., and whether the audio and video codec patent questions ever get settled. If little positive happens on those issues then widespread adoption of HTML5 et.al. will be significantly delayed, at least in the desktop space (though I wouldn't discount what happens in the space of mobile devices).

Final notes: 1) I do agree with you on SVG, which was/is a classic "everything but the kitchen sink" standard. In the non-Flash space I think Canvas will displace SVG almost completely, especially if Canvas 3D takes off. 2) Other folks interested in getting a flavor of the Flash vs. HTML5 debate can check out this article and the articles it links to.

[Jeff Harrington]

One more quick thought about Flash. In many ways, the web is becoming interactive TV (I used to work for an interactive TV company) and that means one thing, Flash. Video now is Flash. That'll keep Flash going for a longggg time.

:-o

Frank Hecker said:

Jeff: I agree that Flash is the best approach for now. HTML5 at this point is a bleeding edge technology, and my point was to illustrate what's becoming possible with open web technologies, not to urge that people abandon Flash wholesale.

What happens in the long run is an interesting question. I'm of course biased given my work for Mozilla, but I suspect that there's going to be a *lot* of innovation around HTML5, Canvas, and other web technologies, and between advances in those technologies and the gradual irrelevance of older browsers (e.g., IE6) I think Flash does face a serious long-term threat to its dominance. The key questions include how fast Microsoft implements HTML5 et.al., and whether the audio and video codec patent questions ever get settled. If little positive happens on those issues then widespread adoption of HTML5 et.al. will be significantly delayed, at least in the desktop space (though I wouldn't discount what happens in the space of mobile devices).

Final notes: 1) I do agree with you on SVG, which was/is a classic "everything but the kitchen sink" standard. In the non-Flash space I think Canvas will displace SVG almost completely, especially if Canvas 3D takes off. 2) Other folks interested in getting a flavor of the Flash vs. HTML5 debate can check out this article and the articles it links to.