proof of concept for hyperaudio notation

For Music Hack Day in San Francisco this past weekend I did a hack related to my blog post on “hyperaudio notation”. My idea was to caption a recorded song using music notation, as an instantiation of ideas like hyper video, hyper audio, popcorn.js, and WebVTT.

There is a recording and a score. The recording is an MP3, the score is a PNG. The purpose of the system is to move a highlight through the score in sync with the MP3, so that the listener can see which part of the notation in the image is currently being played. It’s like text captions for a person talking.

I could have designed it to show just a portion of the overall score, but showing the entire image with a moving highlight was easier.

To move the highlight in sync with the music, you train it. Pushing a button marked “start recording” initiates a training run. The music starts, in time with a recorder for clicks within the image. When you click in the image the time and location are recorded. The trainer clicks in the image in sync with the music. When the first bar is played, click on the first bar in the image. Continue until you have provided music captions for as much of the song as you want. Then press “stop recording.”

At this point, press the “play recording” button to rerun the training session.

The vision is that the training would be done by the person publishing the page, and visitors would just use the “play recording” button.

To see it in action, go to the live demo code or view a screencast I made. (The live code was super quick and dirty and assumes that you have exactly the same everything I do, including browser, bandwidth, etc. Chances that it will actually work are slim).