Data Visualization in MusicJason ForrestBlockedUnblockFollowFollowingJan 28“Giant Steps” Michal Levy, 2001Last fall I went to an Edward Tufte lecture where he began with a very effective, very sweet video of music visualized in a sequence:Tufte knows this is a great and charming start to a lecture.
He knows it provides a welcome change from the outside world; an elegant fusion of music, color, and motion that perfectly introduces the representative abstraction that is data visualization.
He says “It makes the eyes as well as the ears listen” and explains the scrolling rectangles that light up in time with the music.
It is as much a visualization of how music can be anticipated as it is a visual representation of the notes.
It is art and performance and Tufte nails it.
I’m a musician, so this looks very familiar.
The video is basically a midi-sequencer, animated with the notes being highlighted.
It’s not a terribly novel concept as it is literally the basic pattern that many electronic musicians have been using since “the old days” as you can see in this Impulse Tracker video originally from 1998.
These screenshots of “tracker” software show the numbers scrolling vertically while the music plays.
The tones are entered in the panel in the middle, as {note}-{value} (ie:E-7), so what you see is both instrument and the visualization of the playback.
Super standard stuff; you can find a zillion of these on youtube.
A classic Piano Roll UI from Music Studio ProAs computers got faster and less expensive, MIDI was introduced, and with it, the Piano roll became the standard.
The vertical scroll of the tracker didn’t fit with our cognitive understanding of the horizontality of the timeline.
The Piano roll rotates the tracker 90 degrees so that the sequence is now optimized for the horizontal screen.
The pattern stuck and the concept of notes (or audio clips) presented as horizontal bars became the standard design pattern for most Digital Audio Workstations (DAW).
With this in mind, it’s exciting to consider just how differently Michal Levy approached her 2001 visualization of John Coltrane’s Giant Steps.
Certainly, the Jazz aspects come through:Youtube was created in 2005, so if you’ve seen this before it’s likely that it was before the luxurious HD streaming that we have now.
Luckily, Levy has uploaded a new version above with delightful sketches and background notes on her personal site.
“I translated Coltrane’s mathematical approach to architecture.
His musical theme defines a space and the musical improvisation is like someone drifting in that imaginary space.
”Levy does way more than just step through the notes.
Her single pixel becomes a cube that becomes a skyscraper on Coltrane’s first pass.
Then, as Coltrane rips into a nimble solo, the pixel adds form, color, and depth to each floor in what becomes a house.
A squealing note becomes an elevator shaft, each staccato note a blue window and suddenly the rectangles form a cityscape.
On the last pass through, Levy’s structure is released, each free multi-colored block forms a cloud of color, leaving us finally with our initial single block.
Coltrane’s music is both represented and visually performed by Levy.
The transcendent nature of music inspires the artist to make something beyond reporting the notes, making something entirely more inspiring — even moving.
Of course, there’s just so many fantastic videos that visualize music in novel ways.
IDM heart-throbs Autechre have long paired generative audio with generative video, sometimes even creating a single extremely complicated Max/MSP patch to perform both simultaneously.
While this isn't one of those, the 2002 video for Gantz Graf is (IMO) probably the best pairing of Tron-inspired / Warp records aesthetics mapped to each stuttering beat.
While we’re here in the IDM world, I need to also post this video by the incredible French video cooperative, Pleix.
It uses the language of data visualizations for, um, other… NSFW purposes.
Back on track again, with the advent of more advanced 3D rendering filters and methods, the simple sequence that we saw at the top of the article can take on a more advanced form.
Here’s Ólafur Arnalds’ video from 2009 showing how the same, simple sequencer concept can be greatly elaborated on.
This time the instruments are personified by multicolored smoke apparitions.
The notes themselves vivid and multicolored, the movement mapping the pitch of the melodies to movement.
Once a note is performed it desaturates and drifts away.
It’s a visually interesting video and shows how a simple concept can be repurposed into a very different form.
An old friend of mine from the music days is Mr.
Gregg Gillis, aka Girl Talk.
He’s probably the most famous mashup artist in the world.
His music (like mine) is a combination of other songs edited into a seamless collage that becomes categorically new and different from the sum of its parts.
Here’s a very basic video showing what songs are used in a single song.
But that’s literally just an illustration.
The REAL story is this amazing visualization by Phil Gedarovich that explores a Girl Talk track and maps the artist, type, country, genre, and popularity of each sample.
Then, as the mashup plays, Gedarovich zooms in and out of the overall timeline, showing how each is layered.
Time shows up in the background or even just zipping around the virtual canvas.
While it might miss the cool-factor, its visually interesting and illustrates and informs the data-density of the music.
It’s hard to get more stylistically different from Girl Talk than the classy video for the “Bruises.
”Created as a personal collaboration between the virtuosic and highly unique guitar of Kaki King and the virtuosic and highly unique data visualizations by Giorgia Lupi, “Bruises” takes the manually collected daily observations of a family wrestling with a very scary illness and transforms it into multi-media art.
Lupi writes (and speaks) about it far more eloquently than I can, so please read her fantastic piece on this project.
She ends her article with a quote that I find very inspiring — even moving:“We believe though, that radical experimentations of this kind, artistic experimentation of this kind, can be a starting point to reconsider what we define as data in the first place.
”I’d argue that the converse is true as well.
If the visual beauty of the midi-sequence can be understood as a data visualization, why can’t we explore the audio qualities of the output of a sorting algorithm?A truly satisfying experience filled with drama and release; this video is the Marie Kondo of theoretical sorting algorithm analysis.
Crafted as a demo for undergraduate computer science students, Timo Bingmann’s video became an unexpected viral sensation.
A top reddit comment proclaims “I feel like this is exactly what people in the 1950s would think people in 2016 would watch as entertainment.
” to which someone replies: “they weren’t wrong.
”A blog post about the technique and it’s unexpected popularity on a gloriously brutalist website further compliments the experience.
Bingmann’s shock is still evident six years after the fact.
While basic, the video sounds quite interesting and pairs with the visual to make something special.
After the first messy group of lines is organized into an even gradient and the green-hued low-high sweep signals the job done, the viewer is clued into the fun to be had.
With each successive pass, we see the different methods of organization, the different logical approaches to tackling a messy problem.
Color takes on the role of helpers, categorically organizing into subgroups to get the job done, all the while monitored by the ever-present din of a rising or falling tone.
It’s a nail biter who’s purpose is to resolve itself in a way that sparks joy.
And it seems like we just really, really, really like repetition.
This video by Vox is full of interesting visual representations of music.
Vox’s “Earworm” series is totally worth your time.
In fact, my entryway into the series was their fantastic description behind “the most feared song in Jazz” — Giant Steps by John Coltrane.
So there we have it.
Why not add your favorite data-inspired music videos in the comments?.Let's see how many we can collect.
.