This is basically the reason why video sources are so heavy, by the way.
Converting Uma Thurman from frames to RGB spreadsheets.
Frames from Pulp Fiction, by Quentin Tarantino.
[©Miramax]Once the processing pipeline has run, we end up with a comma-separated values file of three columns (i.
e.
RGB average values) and N rows, one for each frame of the input video source.
Now we are ready to go!II.
Get the hands dirty: framelinesFramelinesSince our input is basically just a temporal sequence of colors, the very first thing one might plot is a color timeline.
All the colorful barcodes we’re going to create from now on will be referred as framelines.
Beautiful pieces of art based on movie framelines can be found here (many thanks to Charlie Clark for the inspiration!) and here.
This is an example of what we’ll be able to build at the end of this chapter:Frameline representation of 2001: A Space Odyssey, by Stanley Kubrick.
Each line represents the average color of a consecutive set of frames, starting from time 0 (left) to the end of the movie (right).
The last tile is the average tone of the entire movie.
The thickness of tiles depends on the time window we want to average.
If the time window is too wide, meaningful color information can be lost in the averaging process; if we pick a very short time window instead, overfitting will make the frameline quite difficult to interpret.
If we juxtapose the framelines of Miyazaki’s Princess Mononoke and Nausicaa, we can clearly see how the most interesting color variations are washed out when the sampling window is too wide.
Princess Mononoke and Nausicaa of The Valley of The Wind framelines.
In both the movies, blue scenes start to fade out when the sampling window exceeds 100 seconds.
As the time window become wider, the main color of the movie kills all the other tones, starting from the less represented ones.
Framelines implementation in R ⑇Let’s see how to implement all of this in R now.
We already have a .
csv file with the average RGB value of each frame, so the data import is pretty much done.
Now that we have a proper input data frame, we will use the ggplot2 package to draw our first colorful frameline.
Ggplot is probably the most powerful and popular data viz package on R.
Plotting complex framelines requires a bit of mastering with ggplot and fall ouside the purpose of this first article.
Anyway, here is a simplified but fully-functioning script to plot your first framelines while you wait for Part II:Color averages usually appear to be very dark/desaturated.
This is quite physiological, as scenes are often played on neutral shades of grey and dark tones, and today’s movies tend to be a bit colorless.
We can help our brain and eyeballs in dinstinguish colors with a little vividness boost.
This can be easily done switching from the RGB color model to the Hue-Saturation-Lightness model (HSL) and turning up the last 2 components.
We will spend some words about HSL later.
For now, let’s just say that the modCol function in plotwidgets package makes the vividness boost implementation very straight-forward.
Framelines inspection: the Star Wars sagaNow, let’s make things more interesting by inspecting the colors of the Star Wars universe.
With a slight modification of the previous script, we can easily build a visual map like this one:The Star Wars Saga framelines, starting from first trilogy (1st-3rd row) to Episode VIII (last row).
Apart from being a piece of art I would proudly hang above my bed, this collection of framelines discloses a significant amount of information about George Lucas’ use of color in his epic space opera.
Let’s focus for instance on the episode with the more atypical palette: The Empire Strikes Back (2nd row).
The Empire Strikes Back frameline with some manually extracted key frames.
[©Lucasfilm LTD]In this episode of the Skywalkers’ odissey, our heroes cross the galaxy landing on three different locations: Hoth, Dagobah and Cloud City.
Here George Lucas sapiently associates different dominant colors to each location: Yale blue for the icy Hoth, a dark seaweed green for the swampy Dagobah and a pastel violet for Cloud City.
Different colors help to show the spatial transition along the story, but they also set the tone of each section tweaking with saturation and brightness: Luke’s introspective moment on the swamps of Dagobah is less saturated then the sorrounding sections, and every time the Dark Side of the Force rises, dark tones become dominant.
These patterns can be observed several times along the entire saga, giving it some color consistency.
The most vivid and intense point of the entire saga.
Can you spot it on the framelines map?.[©Lucasfilm LTD]Color consistency can be observed whithin each trilogy as well: dominant hues are cold in the Original Trilogy, switching to a much more warm tone in the Prequel Trilogy where the climax peaks on an intense fire red during the famous duel on Mustafar.
Let’s try to visualize these trends.
Summary Tiles: Hue, Brightness and SaturationFew sections above, we mentioned that RGB isn’t the only color model around, neither the most interesting one for our purposes.
It’s just easy and intuitive.
To inspect interesting color trends, it may be a good idea to switch from RGB to the HSL model.
Hue (H), Lightness (L) and Saturation (S) are in fact some of the most important properties of color when we speak of chromatic storytelling.
Bicone representation of the HSL color space.
[from Wikipedia]The HSL color space can be seen as a bicone, where each point (i.
e.
color) is expressed in a cylindrical-coordinates system whit S ∈ [0,1], L ∈ [0, 1] and H ∈ [0°, 360°].
While we all know what saturation and brightness mean, the definition of hue may be a bit ambiguous.
The International Commission of Illumination (CIE) defines hue as the degree to which a stimulus can be described as similar to or different from stimuli that are described as red, green, blue and yellow.
We can basically refer to it as the color shade.
Each of these three channels/properties can be isolated setting the others to a fixed value: for instance, we can inspect the average saturation value of each episode by setting the brightness channel to 0.
5 (half intensity) and the hue channel to 0° (i.
e.
pure red).
Brightness and saturation channel might be re-scaled to exhalt the differences between movies.
Such inspection can be visualized in form of summary tiles, where each square represents a different episode.
Summary tiles for saturation (left), brightness (middle) and hue (right) channels of the Star Wars saga.
From top to bottom: Sequel Trilogy, Prequel Trilogy, Original Trilogy.
The above shown summary tiles confirm what we said above about hues.
Moreover, we can clearly see that the Prequel Trilogy appears to be less dark and highly saturated compared to both the first and the latter one.
On the contrary, the first two episodes of the Sequel Trilogy are the darkest and less saturated chapters of the entire saga, with Episode VII: The Force Awakens winning the first prize in both the categories.
HSL channels trend over time can be better inspected merging together all the framelines and smoothing the resulting time series.
Saturation (left), brightness (middle) and hue (right) channels trend over time in the Star Wars saga.
At this point, it could be funny to start guessing how the next episode will be.
Considering that the 1st and 3rd chapters of each trilogy have always been less saturated than the 2nd, we could expect from Episode IX a lower level of saturation.
Concerning hues, the color tiles matrix is pretty much symmetrical, therefore it’s hard to tell what the temperature of the movie will be; considering the overall tone of the saga, we can speculate on a not-so-warm red-ish overall tone.
Place your bet and see you at the end of December to know who’s the winner!III.
Have some fun: movies and directors comparisonColor fingerprinting: Wes Anderson and Hayao MiyazakiNow that we know how to collect the hints that might lay within the color of a movie, let’s have a look on the work of some of the most iconic directors and their movies.
Many filmmakers have developed a distinct aesthetic style that is easily recognizable through the use of color palettes in their works.
Among recent movie directors, one of the most noticeable color palette belongs to Wes Anderson and his extravagant stories.
Fantastic Mr Fox, by Wes Anderson.
[©20th Century Fox, palettes from here]With a great love for pastel tones, Anderson puts these kinds of colours into almost every element of the scenery.
This allows him a complete monopoly over the meaning and subtext of his films.
Color fingerprints of three contemporary directors: Wes Anderson, Hayao Miyazaki and Christopher Nolan.
There are many conscious and unconscious bias that lead a director to choose similar palettes over his/her career.
For instance, nature and ecology are the milestones of Miyazaki’s mindset, and this is clearly reflected in the color palette he uses in the vast majority of his masterpieces.
Color symbolism: The MatrixSimilarly to what happens with directors, a particular color palette might characterize movies and tv shows as well; some films play on specific colors so much that they end up identifying themselves with the colors in question.
Let’s think about green, for instance.
What is the first movie you associate with green?.And why is it The Matrix?Framelines of The Matrix trilogy, by The Wachowskis.
The Matrix is probably one of the most obvious and famous case of color association in cinematography, as every scene that takes place inside the virtual world is tinted green, taking inspiration from the phosphorous green of old monitors.
Color symbolism is strong and evident in Wachowskis’ work; in fact, green isn’t the only main color we see: when the scene moves from the Matrix to the real world, green tints are entirely replaced by a dark blue palette.
Chord diagrams of color hues transitions in The Matrix trilogy.
The dominance of green fades away in the third episode, where the plot is almost entirely set in the real (blue) world.
The reason behind this choice is not random of course: in color psychology, the yellow-ish green is associated to nausea and sickness, while the blue tone should make the audience feel more comfortable.
In this sense, by setting up Neo’s normal existence as a sick, claustrophobic green, we can appreciate the freshness the real world offers with its blue tints.
Brightness contrast, instead, is used to remind the audience how dramatic reality is compared to the fake and controlled environment of the Matrix.
The Matrix vs Reality.
Color palettes extracted automatically using k-means.
[© Warner Bros]Sliding through the color spectrum: the Harry Potter sagaSpeaking of green: have you ever noticed how green-ish the latest Harry Potter’s movies are?.Well, it didn’t start exactly like that.
Framelines of The Harry Potter saga.
Can you spot the Dolores Humbridge’s office scene?Actually, Harry Potter sums up pretty well all the things we said about color until now.
Characters and locations and clearly defined by the sapient use of different tints: think about the four houses of Hogwarts, the spells and teachers’ offices, for instance.
During the saga, directors explore the entire color spectrum, slowly fading from a bright, warm, red-brown set to a dark, aseptic and cold green, crossing through some blue tints in the middle of the saga.
In this sense, it’s interesting to see how color evolves together with the characters, getting colder and darker while following the course of events.
Harry’s character color evolution from The Philosopher’s Stone to The Deathly Hallows.
[© Warner Bros]JK Rowling started it all with a book for children, with kids running and screaming around in a magic world.
To capture this athmosphere, Chris Columbus, previously known for directing Home Alone, decided to use a warm and bright color palette.
As for the books, the story quickly grew and matured, becoming more and more complex and tragic as Harry grew up and became and adult.
This is clearly reflected in the chromatic choices of the succeeding directors, as pointed out by the monotonic descent trend of the lightness component over the saga.
Brightness trend over the entrie Harry Potter saga.
The set slowly become darker and darker, with the only exeption of the Harry and Dumbledore Talk at King’s Cross scene.
ConclusionHere is where the first part of this chromatic exploration ends.
We saw that there are many ways to use color in film, and many different techniques can be developed to explore these approaches.
In the next article, we will draw the focus on the technical part, with many reproducible examples in R and a deeper explaination on how to retrieve meaningful information from color temporal sequences.
Please leave your thoughts in the comments section and share if you find this interesting!Tommaso Buonocore – Scrittore – Towards Data Science | LinkedInView Tommaso Buonocore's profile on LinkedIn, the world's largest professional community.
Tommaso has 3 jobs listed on…www.
linkedin.
com.