The core reason is that, as you can see from the examples above, if you were to write a specification in such detail that it would capture those issues, you’d be writing the software and there is really just no way around this (if, as you read this, you’re trying to bargain this one away, I have to tell you — there is really, really, really no way around this.
Full specifications are a terrible economic idea.
Below I’m going to lay out better economic choices)But here’s where it gets interesting: Every programmer who’s been working in the real world for more than a few months has run into the problems I’m describing above.
And yet… we keep on making these spectacularly bad estimates.
And, worse yet, we believe our own estimates.
I still believe my own whenever I make them.
So, wait, am I suggesting that all developers somehow fall prey to the same, predictable errors in thinking?Yep, that’s exactly what I’m suggesting.
Why You Suck at Making Estimates, Part II: OverconfidenceKahneman talks at some length about the problem of “experts” making predictions.
In a shockingly wide variety of situations, those predictions turn out to be utterly useless.
Specifically, in many, many situations, the following three things hold true:1- “Expert” predictions about some future event are so completely unreliable as to be basically meaningless2- The experts in question are extremely confident about the accuracy of their predictions3- Absolutely nothing seems to be able to diminish the confidence that experts feelThe last one is truly remarkable: even if experts try to honestly face evidence of their own past failures, even if they deeply understand this flaw in human cognition… they will still feel a deep sense of confidence in the accuracy of their predictions.
As Kahneman explains it, after telling an amazing story about his own failures on this front:“The confidence you will experience in your future judgments will not be diminished by what you just read, even if you believe every word.
”Interestingly, there are situations where expert predictions are quite good — Below I’m going to explain how to use these situations to hack your own dev process.
But before I do that, I want to walk through some details of how the flawed overconfidence works, on the ground, so you can maybe recognize it in yourself.
What It Feels Like To Be Wrong: Systems I & II, and The 3 Weeks and 3 Months ProblemIn Thinking Fast and Slow, Kahneman explains a great deal of psychology as the interplay between two “systems” which govern our thoughts: System I and System II.
My far-too-brief summary would be “System I does quick, heuristic, pattern matching thinking and System II does careful, rational, analytical thinking.
”And, crucially, it’s as if evolution designed the whole thing with a key goal of keeping System II from having to do too much.
Which makes plenty of sense from an evolutionary perspective — System II is slow as molasses, and incredibly costly, it should only be deployed in very, very rare situations.
But you see the problem, no doubt: without thinking, how does your mind know when to invoke System II?.From this perspective, many of the various “cognitive biases” of psychology make sense as elegant engineering solutions to a brutal real-world problem: how to apportion attention in real time.
To see how the interplay between Systems I & II can lead to truly awful, and yet honestly-believed estimates, I’m going turn the mic briefly over to my friend Edmund Jorgensen.
He described it to me in an email as follows:When I ask myself “how long will this project take” System I has no idea, but wants to have an idea, and translates the question.
Into what?.I suspect it’s into something like “how confident am I that I can do this thing,” and that gets translated into a time estimate, with some multiplier that’s fairly individual (e.
g.
when Bob has level of confidence X, he always says 3 weeks; when Suzy has level of confidence X, she always says 5 weeks).
Raise your hand if you’ve gradually realized you have two “big” time estimates?.For me, it’s “3 weeks” and “3 months”.
The former means “that seems complex, but I basically think I see how to do it”, and the latter means “Wow, that’s hard, I’m not sure what’s involved, but I bet I can figure it out.
”Aka, I think Edmund is totally right.
(For those playing along at home: my “3-week” projects seem to take 5–15 weeks, my “3-month” projects usually take 1–3 years, on the rare event that someone is willing to keep paying me).
Alright, So Let’s Stop Being So Overconfident!You might be thinking at this point: “Okay, I see where Dan is going: we have to approach these estimation challenges in some manner that engages System II instead of System I.
That way, our careful, analytical minds will produce much better estimates.
”Congratulations, you’ve just invented Waterfall.
That’s basically the promise of the “full specification before we start coding” approach: don’t allow the team to make intuitive estimates, force everyone to carefully engage their analytical minds and come up with a detailed spec with estimates broken down into smaller pieces.
But that totally fails.
Like, always.
The real trouble here is the interplay between the two sources of estimation error: the human bias towards overconfidence, and the inherent uncertainty involved in any real software project.
That uncertainty is severe enough that even the careful, rational System II is unable to come up with accurate predictions.
Fortunately, there is a way to both play to the strengths of your own cognition and also handle the intense variability of the real world.
First, how to play to your mind’s strengths.
When Experts Are Right, and How To Use That To Your AdvantageKahneman and other researchers have been able to identify situations where expert judgment doesn’t completely suck.
As he says:“To know whether you can trust a particular intuitive judgment, there are two questions you should ask: Is the environment in which the judgment is made sufficiently regular to enable predictions from the available evidence?.The answer is yes for diagnosticians, no for stock pickers.
Do the professionals have an adequate opportunity to learn the cues and the regularities?”An “adequate opportunity” means a lot of practice making predictions, and a tight feedback loop to learn their accuracy.
Now, 6–18 month software projects just miserably fail on all these criteria.
As I’ve discussed above, the environment is just savagely not “regular”.
Plus, experts don’t get the combination of making lots of predictions and getting rapid feedback.
If something is going to take a year or more, the feedback loop is too long to train your intuition (plus you need a lot of instances).
However, there is a form of estimation in software dev that does fit that bill — 0–12 hour tasks which are then immediately executed.
At that scale, things work differently:Although there are still a lot of variabilities (more on that below), there is some real hope of “regularity in your environment”.
Two four-hour tasks tend to have a lot more in common than two six-month projects.
You can expect to make hundreds of such estimates, in the course of a couple of years.
You get very quick feedback about your accuracyThe highest-velocity team I’ve ever been on did week sprints, and broke everything down to, basically, 0, 2, 4, or 8 hours (and there was always some suspicion about the 8-hour ones — like, we’d try pretty hard to break those down to smaller chunks).
We estimated those very quickly and somewhat casually — we didn’t even use a Planning Poker style formalism.
At that point, you’re using the strengths of System I — it has a chance to get trained, it sees plenty of examples, and there are meaningful patterns to be gleaned.
And, thanks to the short sprint length, you get very rapid feedback on the quality of your estimates.
Wait, Wait, Wait, Let’s Just Make a Thousand 4 Hour Estimates!How can I both claim that you can make these micro-scale estimates, but somehow can’t roll them up into 6–18 months estimates?.Won’t the errors average out?Basically, although I think the estimates at that scale are often right, when they’re wrong, there’s simply no limit to how wrong they can be.
In math-y terms, I suspect the actual times follow a power law distribution.
And, power law distributions are notable for having no stable mean, and infinite variance.
Which, frankly, is exactly how those big waterfall project estimates feel to me.
You might be thinking: how on earth could something you expected to take 4 hours actually take a month or two?This happens all the time: you go to take some final step in something and discover some hideous blocker which completely changes the scope.
E.
g.
at a recent startup, in trying to eliminate single points of failure from our system, we went to put a load balancer in front of an IMAP server we had written.
Our goal was that when one server machine died, the load balancer would just smoothly fail over to another box, and customers would see no impact.
And that seemed like a 4-hour-ish task.
But when we went to actually do it, we realized/remembered that the IMAP server, unlike all the HTTP servers we were so used to the maintained connection state.
So if we wanted to be able to transparently fail over to another server, we’d have to somehow maintain that state on two servers, or write some kind of state-aware proxying load balancer in front of the IMAP server.
Which felt like about a 3-month project to us.
And there is the other reason that short sprints are an absolutely key piece of all this: they place a hard limit on the cost of a horrifically bad estimate.
Are We All Just Screwed?So what do we do?.Just accept that all our projects are doomed to failure?.That we’ll have poisoned relationships within the rest of the business because we’ll always be failing to meet our promises?The key is that you first accept that making accurate long-term estimates is fundamentally impossible.
Once you’ve done that, you can tackle a challenge which, though extremely difficult, can be met: how you can your dev team generate a ton of value, even though you can not make meaningful long-term estimates?What we’ve arrived at is basically a first-principles explanation of why the various Agile approaches have taken over the world.
.