Visualizing Musical PerformanceD.
Ryan MillerBlockedUnblockFollowFollowingMay 25As a musician and a data scientist, I have been intrigued by the thought of visualizing musical performances.
In this post, I outline how to visualize piano performance recordings from the MAESTRO dataset.
Examples are provided throughout the post.
I layout step by step instructions, with code, for opening, cleaning, and visualizing piano performances in the MAESTRO dataset using Python’s Mido, Seaborn, and Pandas packages.
The post concludes with a visualization of Franz Liszt’s Hungarian Rhapsody № 2 and a recording of the piece so that readers can watch the piece unfold through the visualization.
A link to the full Python script used to create the visualizations is also provided.
Joint density plot of Johann Sebastian Bach’s Prelude and Fugue in D Minor, WTC I, BWV 851The DataThe MAESTRO dataset contains over 200 hours of piano performances from the International Piano-e-Competition.
Participants in the competition perform on Yamaha Disklaviers, acoustic pianos that can also capture and playback Musical Instrument Digital Interface (MIDI) data.
The MAESTRO dataset contains MIDI data from contestant performances as well as audio recordings of the performances.
MIDI is a protocal that allows digital instruments to communicate with each other via ‘messages’.
These messages store information about the type of instrument to use for playback, the notes to play, when a note starts, when a note ends, etc.
Introduction: Opening a MIDI File in PythonTo open a MIDI file in Python, install the mido package.
From the Mido package, import MidiFile.
Below is example code for opening a file in Python with MidiFile.
For the MAESTRO data, this process will create a MidiFile object which will contain two tracks.
The first track contains metadata about the performance (e.
g.
, time signature of the composition).
The second track contains the performance messages.
Joint density plot of Wolfgang Amadeus Mozart’s Sonata in D MajorExposition: Munging the DataThe following steps I used to munge the data to create the visualizations in this post.
Step 1: Isolate the second track and then iterate through the track to extract the messages.
The result should be a list of message objects.
When iterating, omit the first and last messages.
The first message is the program message and the last is a meta message indicating the end of the track.
Neither message provides data about the notes played.
Step 2: Iterate through the message list transforming the messages to strings.
Transformation to strings can be accomplished using the str() function from base python or the format_as_string() method from the mido package.
Step 3: Split the message strings into substrings containing a key and value using .
split(‘’).
This will create a list of lists.
An individual list looks like:The first substring of the list only contains a value (the message type).
In order to convert the data to a dictionary, this substring must be removed.
Step 4: Remove the first substring, store it in a list, and convert the list to a dataframe.
Step 5: Convert the other substrings in the list to a dictionary and then to a dataframe.
The = separates the keys and values in the substrings.
The following code iterates through the lists and through each substring to create key-value pairs.
It then stores these key-value pairs in a list of dictionaries (one dictionary for each list of substrings).
Finally, it converts the dictionary to a dataframe.
Step 6: Concatenate the attribute dataframe (df2) with the Dataframe containing the message type (df1).
Step 7: The time variable in the dataframe represents the time passed since the last message.
The note variable presents the piano key played.
In order to plot the data, a measure of time elapsed is needed to plot the data over time.
A time_elapsed variable can be created using Python’s .
cumsum() method.
The note variable presents the piano key played.
Plotting the note requires transforming the variable from type string to type float.
Step 8: Filter out control messages and note_off messages as these messages will not be used in the visualization.
The control messages indicate when the sustain and soft pedals are pressed and released.
The control messages have a message type equal to ‘control’.
The note_off messages are indicated by messages of type note_on that have a velocity of 0.
Step 9: Drop unnecessary columns from the dataframe.
Step 10: The final step of the munging process is to add a row to the beginning and end of the dataframe.
Adding these rows will provide some space between the edges of the visualization and the data points.
The new first row is assigned a note value of 0 (a value outside the lower range of the piano) and a time elapsed value equal to -0.
05 times the maximum time elapsed value.
(The first note_on message has a time_elapsed value greater than 0).
The new last row is assigned a note value of 127 (a value outside the upper range of the piano) and a time elapsed value equal to 1.
5 times the maximum time elapsed value.
Adding these rows will also make it possible to generate visualizations with smooth edges.
Joint density plot of Edvard Grieg’s Lyric Piece in A Minor, “Waltz,” Op.
12 № 2Development: VisualizingTo visualize the performances, I utilize Python’s Seaborn package.
The x-axis represents time elapsed (with time increasing from left to right) and the y-axis represents the pitch of the note being played from low (bottom) to high (top).
The plots display the joint density of notes over time using hex bins.
Color is used to convey the frequency of notes within the range of pitches and time periods contained within a hex bin.
In the plot above, higher frequencies are represented by darker shades of green.
Th joint density plot displays the frequency of pitches within a range defined by the hex bin and how those frequencies change as the performance progresses.
The result is a plot that visualizes pitch (y-axis) and pitch frequency (color) over time (y-axis).
The plot contains all of the information necessary to follow along with the performance of the piece visualized in the plot (see the Hungarian Rhapsody example below for a demonastration).
The following steps break down how the visualizations in this post were plotted.
Step1: Set the Seaborn style to ‘white’.
This step provide a clean white background on which to build the visualization.
Step 2: Define the plot, the limits of the x-axis and the limits of the y-axis.
Here is where the addition of a first and last row in Step 10 in the munging process helps with the visualization.
The x-axis limits are defined by the minimum time elapsed and maximum time elapsed.
Both of these quantities were set by the additional rows.
The y-axis limits are set to 16 and 113.
The reason for this is to provide smooth edges to the plots.
MIDI note values (plotted on the y-axis) can take on values between 0 and 127.
The piano, however, only has 88 keys which are assigned MIDI note values between 21 and 108.
The note values for the first and last row were set outside of the range of possible values for a piano.
As a result, by setting the y-axis limits within the minimum and maximum note values set by the first and last rows, the plots:Display all possible values for the piano,Do not display the artificial note values from the first and last rows,Include a space between the edges of each visualization and the lowest and highest notes, andThat space is colored the same as the background of the plot.
One additional note about the plot function.
The gridsize indicates the number of hex bins in the x-direction.
Changing the gridsize changes the size of the hex bins and, subsequently, the size of the hexgons in the plot.
The resulting plot for Rimsky-Korsakov’s Flight of the Bumblebee can be seen below.
Joint density plot of Rimsky-Korsakov’s Flight of the Bumblebee (with labels and marginal plots)Step 3: Remove unwanted plot elements.
For the purpose of providing an artistic rendition of the joint density plot, I remove the marginal plots, axis and tick labels, and axes.
(NOTE: Removing these elements is the antithesis of best practice for providing a meaningful visualization.
) The following code removes the axes, the marginal plots, axis labels, and tick labels.
Joint density plot of Nikolai Rimsky-Korsakov’s Flight of the BumblebeeRecapitulation:This post presented steps for opening, cleaning, and visualizing MIDI performance data from the MAESTRO dataset.
Code snippets were provided to demonstrate the steps.
The performance data were visualized using Seaborn’s jointplot method.
The plots visualize the density of pitches (alternatively, the frequency of notes played) over time.
The result is an artistic depiction that captures of a piece of music in a single image.
For those readers interested in seeing how a visualization aligns with a performance of a musical work, below is an example.
The visualization is of Franz Liszt’s Hungarian Rhapsody № 2.
The visualization is followed by a YouTube video of Adam Gyorgy’s performance of the work.
For readers interested in creating their own visualizations, my script and documentation can be found on GitHub.
Joint density plot of Alban Berg’s Sonata Op.
1Coda: Hungarian Rhapsody № 2Below is a visualization of Franz Liszt’s Hungarian Rhapsody № 2 as well as a video of Adam Gyorgy performing the piece.
As you listen to the video, follow the performance through the visualization.
Joint density plot of Liszt’s Hungarian Rhapsody № 2BibliographyCurtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck.
“Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset.
” In International Conference on Learning Representations, 2019.
Clark, I.
(2016, April 25).
RE: MIDI Specifications [Online Discussion Group].
Retrieved from https://www.
midi.
org/forum/228-writing-midi-software-send-note-off,-or-zero-velocity-note-on.
.. More details