By Michael Kaminsky, Data Strategist.
Original.
Reposted with permission.
The landscape of the data and analytics world is shifting rapidly.
In many companies, the roles and responsibilities of data engineers, analysts, and data scientists are changing.
This change has created the need for a new role on the data team which some have taken to calling the “analytics engineer”.
The analytics engineer sits at the intersection of the skill sets of data scientists, analysts, and data engineers.
They bring a formal and rigorous software engineering practice to the efforts of analysts and data scientists, and they bring an analytical and business-outcomes mindset to the efforts of data engineering.
It’s their job to build tools and infrastructure to support the efforts of the analytics and data team as a whole.
Before we dive further into the role, we should cover some background on the “traditional” roles on the data team1.
In the last few years, we’ve seen a lot of exciting developments in the analytics domain that have caused a shift in these traditional responsibilities.
They are:The first two, taken together, have shifted the role of analysts dramatically.
Nowadays analysts must know how to write SQL, use git/github, and generally spend a majority of their time writing code.
While they aren’t necessarily trained as software engineers, they are now responsible for managing substantial codebases.
Similarly, while data engineers used to spend a lot of time split between building new data integrations between systems or working on platforms for scalable computation, most of that work can now be offloaded to Stitch/Fivetran (integrations) or to the warehouse itself (just let BigQuery figure out the optimal query plan).
Finally, data scientists have suddenly become responsible for managing sophisticated production systems that are making real-time productions with significant business impact.
So what are the new roles and responsibilities?Those who have worked in an organization like this before have likely felt the pinch of a missing role.
While data scientists and analysts are writing a lot of code, being great software engineers isn’t what they’ve been trained for and it often isn’t their first priority.
Similarly, while data engineers are great software engineers, they don’t have training in how they data are actually used and so can’t always partner effectively with analysts and data scientists.
I believe this gap should be filled in by analytics engineers.
Their job is to:While they have a lot of strengths, analytics engineers can’t (and shouldn’t) do everything.
Though they may have exposure to analytic methodologies, they often aren’t as strong at communicating results or winning over business partners.
While analysts specialize in deriving insights and communicating those to a wider audience, analytics engineers often don’t do that as well.
Similarly, while a data scientist may have a graduate degree in mathematics and a deep understanding of statistical theory, an analytics engineer will generally favor working code over theoretical correctness (so know what you’re getting into!).
Finally, with such a blend of skills, analytics engineers generally aren’t the strongest software engineers you can find – their value comes from their well-roundedness.
So while you may expect your systems engineers to have a deep knowledge of both networking and CS algorithms, analytics engineers often have shallower and more applied knowledge (and will need support from more technical engineering partners on especially tough engineering challenges).
This role can provide a multiplier effect on the output of an analytics teams.
By helping analysts and data scientists scale their efforts without getting bogged down in unmaintainable code, you can run much leaner.
Similarly, with talented software engineers on the team, analytics teams don’t get blocked waiting on resources from other parts of the technology organization.
Finally, with these resources you have someone naturally ready to partner with the rest of the tech organization on building data-driven products (like adding a recommendation engine into a web platform) than if you just have data scientists and analysts who might be less familiar with the operational constraints of such a feature.
Some readers may be thinking that this role sounds like a real unicorn that will be impossible to hire.
I don’t believe that’s true – many teams have the people with the requisite skills and experience already on their teams today.
Often this person looks like someone who was trained as an analyst or data scientist but who has elected to go deeper into software engineering.
They’re often the person showing new team-members how to set up git, who are volunteering for tasks with thorny technical issues and avoiding anything that requires working excel, or who are taking software engineering MOOCs in their spare time.
There are a surprising number of these people out in the world today, but in the status-quo world they often go under-utilized and under-appreciated.
We as analytics leaders, by recognizing the importance of this skill set and the value of this role, can work to both cultivate and develop analytics engineers by 1) recognizing individuals who are positively inclined to this combination of responsibilities and 2) helping them find the right place in the organization where these skills can be fully leveraged, recognized, and appreciated.
I believe that recognizing the role and the title as an important job that is in fact distinct from the responsibilities of analyst/data-scientist/data-engineer is the first step.
I would love to hear your thoughts and opinions on this role and how to make analytics teams works more effectively.
Please send me an email or join our slack channel to share.
Bio: Michael Kaminsky likes to build teams that build things and is a statistics nerd who somehow isnt very good at math, but a software engineer who isnt very good at writing code.
Original.
Reposted with permission.
Resources:Related: var disqus_shortname = kdnuggets; (function() { var dsq = document.
createElement(script); dsq.
type = text/javascript; dsq.
async = true; dsq.
src = https://kdnuggets.
disqus.
com/embed.
js; (document.
getElementsByTagName(head)[0] || document.
getElementsByTagName(body)[0]).
appendChild(dsq); })();.