You Suck at Estimating (because you're human)

Now that my click-bait headline has landed you here I’ll immediately come clean and acknowledge statistically some of you are great at estimating (and that’s awesome for you)! However, the vast majority of humans on this planet do pretty much suck at estimating, especially long and/or complicated things… Like building software. 👍

Doin’ It Right

There are definitely places in academia or research or highly speculative moon-shot projects where the focus is on experimentation and pushing boundaries instead of timelines, but for the vast majority of us, software estimation matters because we need to get paid!

In order to pay staff a modern SaaS business needs to make money and it needs to move fast. It needs to ship new features and enhancements and refinements and bug fixes fast. Faster than the competition. Faster than some new vibe-coded challenger. Fast enough that customers will see value in the big ol’ cost increase they’ll get hit with every time their contract comes up for renewal . In the complicated planning tug-of-war that happens between sales, marketing, product, operations, and engineering teams, timelines matter and the estimates made which support those timelines matter a lot.

Oh Yeah?

Hard truth 👉 Whether you agree that estimating is important or not is ~ irrelevant. I can almost guarantee you’re being asked to estimate projects or epics or tasks or iterations or some other thing that effectively means “work”. Further, I’m willing to bet your performance is linked to your ability to estimate.
Don’t believe me?
Consider this: Do you feel like you could keep making commitments, missing them, and stay employed or advance your career? I doubt it.

On a more positive note if you’re working somewhere with good development practices then the occasional estimation miss won’t have dire consequences. Those misses will be leveraged as learning opportunities within a feedback loop and they’ll (ideally) drive future improvement for yourself and/or your team.

Human After All

If you’re not a robot and are indeed human, then you’ve got some built-in biases and tendencies which impact your ability to make good estimates. Here are some of the most impactful:

• Underestimating Complexity: We often underestimate the complexity of tasks we’re not familiar with. This is why more experienced and skilled individuals will be better able to estimate tasks for themselves while being much worse at estimating for those less experienced
• Optimism Bias : Most people focus on “happy path” or “best case” scenarios when estimating and fail to weigh risk appropriately when estimating
• Anchoring Bias : Estimations can be skewed by “tunnel vision” - excessive focus on one aspect of a task to the detriment of the whole
• The Planning Fallacy where many of the issues above work together.

Those are the primary reasons we, as humans, suck at estimating but if we broaden scope to include work we can find a few more…

Something About Us

As if being human wasn’t enough of a hindrance, we often make it harder for individuals and teams in software engineering to estimate accurately. Some of the most common problems include:

• Failing to properly prioritize needs within a solution (ex. Providing a huge volume of text laying out requirements vs. MoSCoW format)
• Providing the “what” in requirements without providing the “why”
• Inadequate planning time
• Insufficient task decomposition
• Failing to calibrate estimations by reviewing historical estimates
• Insufficient or absence of collaboration and communication (between individuals, teams, and even orgs!)
• Poor understanding of tools (ex. CI/CD) and/or required processes (ex. compliance)
• Failure to recognize impact of unplanned work

Harder, Better, Faster, Stronger

I’ve outlined the problems which impact estimation accuracy, now I’ll provide some solutions. The following tips and techniques are things I’ve seen make a difference over the years. Some of these are easier to do than others and some are doable at the individual level while others require larger group action. Process related stuff skews toward agile processes, but I don’t think you have to be “doing agile” to benefit from this advice.

1. Break Up Work Into Tasks Which Can Be Completed In <= 1 Week

This one isn’t a hard rule and exceptions will come up. Sometimes it just doesn’t make sense to break something down to work that can be timeboxed to a week or less… But it sure works better when you do!
By keeping commitments short it forces individuals and/or teams to really think about the work that needs to happen. A tight, well understood scope may not eliminate complexity but it’ll ensure it’s understood and a short time period minimises unplanned work interruptions which may occur. A short time period also makes it easier to quickly think about “unhappy path” outcomes and makes it easier to build in buffer to the task. This typically also means task decomposition is happening. If you’re working with total unknowns (ex. A “spike”) a max 1 week timebox is a good way to keep things moving and prevent rabbitholing; shorter timeboxes are better!

2. Don’t Commit to the Estimates of Others

Sometimes this works out fine; most of the time it doesn’t. A whole lot goes into making an estimate and no one should ever feel pressured to commit to an estimate they didn’t give. If faced with this – talk it out! Sometimes a little bit of conversation can get people aligned and the shorter estimate will work out after all.

3. Review Estimation Accuracy

When it comes to reviewing estimation accuracy the biggest mistake you can make is not reviewing it! There are plenty of ways to do this from committed vs. delivered points or task counts to larger epic level management. My personal advice is: keep it granular. The more granular you can track this and surface it to individuals, the better. Empower individuals to own their commitments. Reviewing past estimation accuracy directly counteracts our human trait to ignore this information and it leads to swift improvement.

4. Always Plan Up Front

There are squads out there that are made up of coding rockstars and I’m willing to accept they can just flow the most pure form of agile where there’s no up front product plan or design docs or technical plans or API contracts… But for everyone else?
Make a plan! You don’t need to spend weeks writing product or technical design docs, but you sure as heck should be documenting important things like core product needs (use MoSCoW to keep it simple!) and planned technical approaches. From there, as a team you should then be breaking those plans down into executable tasks, stories, sticky notes, or whatever.
Essentially, you should generate a collection of tasks (for lack of a better term) which map back to your plans and will take less than a week each to do. You then leverage this when estimating the overall deliverable adding in buffer for unknowns + unhappy path results + unplanned work (based on historical rates that you totally know, right?! ) to get something you’ll probably hit.

Sound like madness?
Sound like waterfall-in-sprints?

It’s not! You can still adjust and tweak that plan as you go which is actual agility, but you’ve planned and discussed enough to counteract human and process related issues. 🎉

5. Use the Why to Collaboratively Derive What and How

First, let’s agree that why is the critical justification for doing work because there’s always more that could be done than can be done. The why justifies saying yes to work and it’s what drives prioritization.

In a traditional agile setup Product brings the why. In a highly functional agile environment Product and Eng then collaborate to define the what, and then Eng commits to the how. It sounds simple because it is! In most cases a collaborative cross-functional team can go from why ➡️ what ➡️ how in a few hours/days and then into execution.
And here’s the kicker: In my experience a team planning (see point #4) & executing like this will produce plans with highly accurate estimates.

Consider this:

• By starting with the why alone and then building the what collaboratively it forces a deeper understanding of the problem & solution spaces
• Corner cases, error handling, and other unhappy path experiences get discussed alongside the happy path experience
• Work with little value gets pushed to “should” or “could” or “won’t”
• A deeper understanding of the what leads to more accurate task breakdowns
• Technical unknowns are called out and identified as risks more easily (or eliminated from plans up front)
• The how is often (but not always) discussed in parallel and can be ready quickly after

In short, anchoring bias, optimism bias, complexity underestimation, task decomposition, lack of collaboration, need prioritization/requirement clarity, and the planning fallacy are all largely mitigated by working this way!

This is probably the most powerful change a team can implement to drive estimation accuracy improvement. Kind of funny how something so simple could be so powerful , eh?