Not everyone needs real-time analytics, including you

Then it’s a lot less likely.

The best time to have data and analytics is when you’re making a decision.

I might even go so far as to say that decision-making time (or in the preparation to deciding) is the most important time you need data analytics available.

Notifications from metrics when your systems break is another critical time to have them available (arguably you need to make a decision then too).

Just about everything else is a nice-to-have.

I feel this is the key to deciding how to balance the good, fast, cheap equation.

They key is knowing and identifying what the timing of decision-making and building your metrics infrastructure to meet those needs.

If something changes in your system and you NEED to respond within 5 minutes (or 5 milliseconds), your analytics has to move faster than that to give you time to react.

For such a situation, having the low data analytics latency is of value to you, and you’ll want to invest in systems to keep things speedy.

Meanwhile, if managing sales accounts where every last sales contract takes months to land, and contracts last for years, it’s going to be extremely unlikely that you’re going to benefit for a low latency analytics system, the latency lives somewhere else.

Similarly, if you’re ONLY using your data for quarterly reports, and there’s no actual business use case for doing it any faster, you probably don’t need a fancy system either.

Notice, this is largely an organizational issue, not a technological one.

No amount of speedy analytics can force an org that’s structured like molasses to move like anything but molasses.

Knowing your decision-making cadence helps with costsScaled real-time analytics systems tend to be very complicated beasts.

A certain amount of scale is implicit because (up to a point) you can do fairly low latency analytics with highly tuned and beefy database.

(Yes I’m conflating real-time w/ low latency a bit here.

)Once you hit the limits of what a tuned database can do however, there’s a need for processing streams of incoming data, doing some engineering trickery to bridge the fast short-term data w/ much slower permanent data stores.

You’ll need more computing power to handle the growing data volume, you’ll need more network and storage, then people to maintain the systems.

It’s a significant investment and you pay a cost in maintaining all that machinery.

Meanwhile, batch jobs can be much simpler, at the extreme a single weak analytics database, a SQL query, and a cron script that sends an email can be all that’s needed to deliver a report on time when you have 8 hours to run the job.

But what about interactive exploration?.We need need the low latency for that.

Look, I’ve been an analyst, I get that it sucks to have to wait 10k+ seconds (~2hr 45min!) for a large query of death to come back.

But how much money is your organization willing to spend to get that 10k seconds down to 5k seconds?.100s?.1s?Thanks to diminishing returns, the cost tend to rocket upwards as your time requirements get shorter.

There’s usually some low hanging fruit like upgrading to SSDs or newer hardware.

But no architecture scales forever, so eventually you’ll have to rip everything apart and put together something new, and that’s costly.

So if there’s a business case for it, yeah, fight for the lower latency you need to do your job, to the point where the boss is willing to pay for it.

What if we don’t know we need a real-time analytics system?YES, this is definitely a thing that could happen.

Having more timely analytics can lead to a fundamental change in how the organization operates and all sorts of wonderful unicorns.

But access to tons of noisy up-to-date data can also be a distraction.

It’s hard to predict from a pure tech point of view what you should do here.

Instead, I’d take a step back.

This is a question of about the organizational structure itself.

Is it willing, or even capable, of changing if given access to data faster?.Are there people in place who have an understanding of how to use the data to make decisions, and can change processes to match?.Maybe there’s a use case example from a competitor that they want to imitate?Throwing tech at an organization that isn’t ready to make use of it rarely works well.

So if you’re budgeting to upgrade your analytics infrastructure faster, you might have to budget extra for training on top of it.


. More details

Leave a Reply