Why are Estimates Often Wrong: A Response

This article, Why are software estimates regularly off by a factor of 2-3 times? showed up in my Twitter feed today It was billed as one of the best explanations of the "futility of software estimation." Go read it now or the rest of this won't make much sense. I'll wait.... ok, now that you're back... While I'm generally in agreement with the #NoEstimates folks (movement?), I don't agree with that description of the problem as being the root cause as a general rule. Yes, it might, maybe even probably, applies in the context in which it was written - start ups. That doesn't include most of us. When I push back on giving an estimate, using that as my defense is going to get me laughed out of the room.

The article lays out a fictitious trip between SF and LA with the narrator looking at a map and using a basic rule of thumb on velocity and the "as the crow flies" distance between SF and LA to come up with an estimate of 10 days to make the trip. The basic premise of the article is that we can't estimate what we don't know. That is absolutely true. But there are some other things that are true as well, especially if you can't avoid making the estimate (and I agree, working in a way that makes estimates unnecessary is the ideal, it's just not always possible).

Here's one truth. You should find someone who has made the trip, or one like it. Have them tell you how long they think it will take. Better yet, have someone who has made the trip several times, with different people, under different conditions. Even better find several such people - with different backgrounds (skills). Put them all in a room to talk about their experiences as it relates to this trip, then have them all come to consensus on what they think it will take.

Here's another. If you truly can't find anyone with experience with a particular journey: don't give an estimate on how long the whole journey will take until you've actually done part of it. In particular, if you can see some spots on the map that look like they might be difficult, do some (not necessarily all) of those parts. Then, revisit your estimate. Ok, you still won't be accurate - but you should be more accurate. At the very least, you'll be more likely to over-estimate than under-estimate if you base it on experience with what you think are the difficult parts.

Another truth. There were a lot of unstated assumptions in that story. The road is flat. The road is smooth. There are no obstacles. No one will need a break during the trip. Anyone who has done any hiking before - in CA or not, along the coast or not - will be able to tell you that those assumptions are unrealistic. Expose those assumptions and they become readily apparent. If the narrator had told his friends how long he thought it would take AND the conditions under which he was making his estimate, I'm guessing the friends push back on the estimate as unrealistic. At the very least, you've given yourself (and them) the ability to evaluate your assumptions and test them.

Finally, though I could probably go on, if you give an estimate that allows you to mark a particular day on the calendar instead of range of days, you're treating your estimate as a measurement. Estimates should be ranges, not numbers. They should also be held loosely. Yes, go ahead and make plans based on your ranges, but expect to adjust those plans as you gain experience and improve your estimates on the remaining work. It should not come as a surprise that your plans have to change.

In my experience, there are many reasons why estimates are wrong - failing to account for the unexpected is only one of them and shouldn't be the primary reason. People with experience usually factor that into their estimate. The one unexpected that you can't account for and which will trip you up the most is when the destination changes mid-trip. When you are faced with a situation where you're not sure where you are going - and start-up land or new product development can be one of those places - avoid estimates for anything longer than the immediate work in front of you. If you can live without them entirely, do so. If you can't, then use experience, spikes, exposing assumptions, and ranges rather than numbers, to help make the necessary estimates and use them appropriately.