The snapshot images that proved that horses do leave the ground, and sparked the invention of the motion picture camera.
How To Query the FutureA new physics of business intelligence, and thinkingMark PalmerBlockedUnblockFollowFollowingMar 20BI hasn’t had a major breakthrough in a decade.
10 years ago, visual analytics pioneers ushered in a new era of business intelligence with tools that connected data at the speed of thought.
A $16B software market was born.
But since then, improvements have been incremental.
Location analytics help put data in a geographical context.
NLG augments graphs with text, like movie subtitles.
NLP helps analysts find stuff faster.
Embedded data science helps put machine learning at your fingertips.
These technologies are great but they don’t change how we think, because the underlying physics of BI tools haven’t changed in a decade.
That is, all BI looks at data at rest, not in motion.
A BI innovation called Streaming BI flips this old model on its head and lets analysts ask questions about the future.
The shift is as profound as the shift from still photography to moving pictures.
The Bet That Led to Motion PicturesIn 1872, Leland Stanford hired photographer Edward Muybridge to help him settle a bet that horses leave the ground as they run.
It took six years, but Muybridge perfected a method to take a series of snapshots from multiple cameras, tripped by thin pieces of line broken by the horse.
His snapshots revealed what the human eye could not see: horses do leave the ground as they run.
The evidence is slides 2 and three above.
Stanford won his bet.
But more importantly, Muybridge’s work helped inspire the motion picture camera.
His technique of stringing snapshots together, for the first time, captured a sense of motion.
In 1892, the first motion picture camera was created.
Existing BI tools are built for snapshots too.
They allow humans to analyze data at rest, like a still photograph.
Time-series databases help string together data points so users can analyze time.
But fundamentally, you’re looking at what already happened.
Wall Street Asks to Analyze the FutureFive years ago, “flash crashes” were an alarmingly common occurrence on Wall Street.
A flash crash happens when high-frequency trading algorithms overreact to a stimulus, like a news event.
An algorithmic chain reaction can lead to massive financial losses, quickly.
For example, in 2012, one firm lost $440M in 45 minutes.
To a trader, that’s like losing a Lamborghini every 3 seconds (specifically, a 2019 Aventador at $500,000).
Several banks began to look for a new approach to detect and stop flash crashes.
With everything moving so fast, they needed a new type of BI — something that helped them ask questions about the future, and react before a flash crash happened.
Or, as one executive put it —“I need a live data warehouse where I can ask hundreds of questions about the future and be informed in real-time when those conditions are about to happen so I can act before it’s too late.
” — Electronic trading VPThis was the “Stanford bet” for BI.
We needed to find a way to see what be seen with available technology.
Real-time pioneers set out to answer Wall Street’s “query the future” challenge.
Of course, you can’t literally query the future.
But, by combining real-time streaming analytics with a new technology called continuous query processing, we could create a way to ask and answer questions about future events.
And, by embedding machine learning models inside the continuous query processor, we could apply predictive algorithms in real-time as well.
Let’s examine Streaming BI at work in an extreme real-time environment — Formula One racing.
Streaming BI in ActionThe short movie clip below shows a moment in the life of Streaming BI as it analyzes IoT data flying off sensors embedded in a Formula One race car.
Like a movie, a business analyst can monitor the car’s position in real-time.
All data on the stream of IoT sensors can be analyzed: throttle, RPM and brake pressure are shown here, but hundreds of factors could be analyzed.
Streaming BI reveals what static snapshots could never reveal: motion, direction, relationships, momentum.
It’s like a live surveillance camera for data.
Streaming BI at work: Query the future.
But real-time dashboards have displayed live data for years; what’s different about Streaming BI is how queries are processed, and how the user controls what gets asked.
In this example, when the user creates a map chart for the car’s position data, the BI tool registers this continuous query:Select Continuous * [Location, RPM, Throttle, Brake] from IoT-StreamThe continuous query processor registers the query and pushes any changes to the visualization.
You just set it and forget it.
Computations change.
Relationships change.
Visual elements change.
This example queries the future by continuously answering “When the car moves, show me where it is.
” The system not only shows you where the car is, but it also shows throttle data by the size of a green bubble; the red bubbles show when, where, and how the driver is breaking.
It’s an immersive BI experience, like a movie.
Thousands of questions about the future can be registered: show me when the driver takes a suboptimal path into a hairpin turn; show me tire wear so I can predict and decide when to change them; show me when the weather forecast changes and how it may affect performance.
And because continuous queries are built into the BI tool, it’s as easy to query the future as it is to query an Excel spreadsheet: you just open a connection to streaming data, create some charts and off you go.
AI can’t apply human judgment and reason.
Image from the University of Wyoming.
On Balancing Human Judgement with Real-Time AwarenessAs world poker champion Annie Duke explains in Thinking in Bets, computers and humans are good at different types of decisions.
AI can’t apply basic human judgment.
For example, they’ll mistake an image of a turtle with whipped cream on top as a Cappuccino.
On the other hand, human beings can’t correlate massive amounts of data and apply math on that data in real-time.
The second breakthrough of Streaming BI is that it helps balance the decision-making strengths of the computer and the human: algorithmic decisions from the computer and judgment from the human being.
That is, Streaming BI helps supplies algorithmic inputs to the human.
Like your nervous system, it’s always on guard, checking every stimulus, and telling you when something good or bad happens, algorithmically.
But then it’s up to the human to decide what that data is really saying and to judge based on our uniquely human powers of judgment and generalization.
For example, when our Streaming BI tool detects that our car is gaining quickly on the car ahead, the race strategist must put it in context by understanding who’s driving the car ahead, how the driver is feeling, and which overtake strategies usually work.
This requires the ability to correlate live insights with historical and reference data.
Here’s how it works.
The continuous query “Tell me when my car is in position to pass,” gets triggered.
The attention of the race analysts is drawn to a look at a BI visualization, perhaps by a text message.
At this point, human judgment can occur via a traditional BI tool like Tableau, Power BI or Spotfire.
The human uses the algorithmic event to explore data about the car, driver, and race, and decide how to act.
So Streaming BI, combined with traditional BI, balances human understanding with algorithmic insight.
Cool!.What Do I Do With This?When I show Streaming BI to business people, they usually say: “Cool!.But what do I do with it?” Here’s how to think about the new problems it can solve.
What questions would you ask if you could query the future?The first step is to think about your data that constantly changes: sales leads, transactions, mobile apps, customer service calls, moving vehicles, robots, kiosks, social media activity, websites, customer orders, chat messages, supply chain updates, file exchanges, customer complaints.
Now, think of questions that start with the word “when.
” These are questions about the future: Tell me when a high-value customer walks in my store.
Tell me when a piece of equipment shows signs of failure.
Tell me when a plane is about to land with a high-priority passenger about to miss their flight.
Examples of Streaming BI ApplicationsMany of the most valuable Streaming BI use cases are known because custom real-time applications have been around for a while.
But now analysts can create them on their own with self-service Streaming BI.
For example, airports in Rome and Melbourne use streaming data from reservations, check-in, security cameras, boarding.
They combine those streams with IoT data from autonomous vehicles, planes, and maintenance equipment.
And they can correlate streaming data from retail operations as well.
Streaming BI queries include asking when security problems are about to happen, when baggage gets stuck and when congestion is about to happen.
Streaming BI democratizes real-time technology because, like moving beyond snapshots to moving pictures, you can see what was unseeable.
Retailers and retail banks watch for when important customers respond to a promotion or price change.
They ask about future transactions that could be fraudulent and apply human judgment to decide and act accordingly.
Supply chain, logistics, and transportation firms analyze thousands of connected vehicles, containers, and people in real-time.
Querying for future issues help them to optimize deliveries, detect and fix routing problems before its too late.
Smart City operational teams can monitor streaming traffic data, buses, trains and plans to avoid congestion and ensure rapid response to emergencies.
Energy companies like Anadarko monitor IoT-enabled industrial equipment to avoid production problems before they happen, or predict the best time to perform maintenance tasks.
ConocoPhillips says that systems could lead to “billions and billions of dollars” in savings.
And those flash crashes on Wall Street?.Ever notice how you don’t hear about them much anymore?Snapshots Aren’t Dead.
They’re Just Old NewsSnapshot BI isn’t dead.
Like the still photograph, they continue to be essential for reports, forecasts, and monthly statements.
But Streaming BI is revolutionary.
It’s an opportunity to change your point of view — to query the now, and the future.
Now anyone can see what wasn’t visible before — a new type of business intelligence that can separate digital innovators from digital dinosaurs.
Mark Palmer is the SVP of Analytics at TIBCO software.
As the CEO of StreamBase, he was named one of the Tech Pioneers that Will Change Your Life by Time Magazine.
.