By Tarmo Tali, VP of Engineering, Data, PipedriveWe love data at Pipedrive.
From the boldest business decision to the smallest UI tweak, we want to have just the right mix of data and intuition helping us.
Pipedrive has been investing for years into data processing – here are the 2020 predictions from our data team.
Great Data Science needs Great Data EngineeringOK, you did it.
You hired a bunch of statistics Ph.
D.
s with a high Kaggle score.
You should be well on track churning out great machine learning solutions.
Except, there is a tiny problem with .
“data.
” You know for sure you have it.
You have petabytes of data.
But it looks like its not the right data.
You need labeled data.
Cleaned, profiled, and analyzed data.
Shaped into a form suitable for machine learning algorithms.
It slowly dawns that machine learning is 80% of data engineering.
So besides a bunch of data scientists, you now need a four times bigger group of data engineers.
Data breaks out from boardroom bar chartsAggregating company data into a central Data Warehouse has been serving mostly analysis functions.
But data is useful everywhere; you want your customer service rep to have data on full customer history and the last ten activities.
Data Warehouse has that data.
You want your microservices to know where in the onboarding flow the customer is.
Data Warehouse has that data.
You want to know which product to recommend.
Well.
You know who has the answer.
Providing operational support is a growing role for Data Warehouse.
Peak snake oil in AIDuring the gold rush, sell shovels.
In 2019, not a week passed without contact from the company offering AI services.
They come in all shapes and sizes and provide everything from consulting to “5-minute integration, complete solution” services.
While there are companies who are providing valuable machine learning services, there is also plenty of snake oil out there, trying to cash in the hype.
Be sure to kick the tires before you open your wallet.
The rise of the Chief Data OfficerThere is a thorny issue on the location of the data team in the org chart.
Where does it belong? Is it finance? Marketing? Engineering? Business Operations? No one knows for sure.
It looks like we are coming to an agreement that data should be under data.
There should be a Chief Data Officer who has a seat in the executive team and who takes care of a precious asset called data.
We got Big Data; now its time for Fast DataWe are restless.
The pulse of the time is an all-time high.
If something happens, we need to know now! Yes, its cheaper and more comfortable to load new data during your overnight loading window, but the business never sleeps.
Realtime data is here to stay.
Beauty will save the worldThere are capable visualization tools that still look like they stepped out from Excel 3.
0.
Im looking at you, matplotlib! While capable is good, and correctness is mandatory.
Time and again, flashy beats correct and useful.
Lets make our visualizations beautiful.
People like beautiful.
Mobile BIWe need our profit charts on mobile too.
Right next to our LinkedIn or TikTok app.
It has been taking way too long for technology providers to get there.
Probably because its very hard.
But we are getting there.
Both Tableau and Power BI have capable mobile client and development tools available.
Everything is ready.
Hopefully, you can read your daily business performance report instead of the back of the shampoo bottle while away from your desk.
Non-trend: Self-service BINope.
It has not happened in the past 20 years.
Not going to happen next.
Every year, its just around the corner.
A little bit more and everyone can ask all the questions in their mother tongue, and the system spits up answers.
Look, there is a reason why a good analyst is worth its weight in gold.
Answering questions like “What was the profit last quarter in Germany?” is not self-service BI.
Its a bar chart with extra steps.
Understanding the domain, finding hidden patterns in from a large amount of data, and presenting complex results in an easy to understand way is still far away from machine capabilities.
Bio: Tarmo Tali is VP of Engineering, Data, at Pipedrive.
Related: var disqus_shortname = kdnuggets; (function() { var dsq = document.
createElement(script); dsq.
type = text/javascript; dsq.
async = true; dsq.
src = https://kdnuggets.
disqus.
com/embed.
js; (document.
getElementsByTagName(head)[0] || document.
getElementsByTagName(body)[0]).
appendChild(dsq); })();.