[Oct. 4, 2013 – Update Note: changed link to the Flyvbjerg paper thanks to Dave Timmermann’s comment.]
Note: I do not want to debate the usefulness of estimates in software development projects, as that’s a completely separate discussion. For the sake of this thought experiment, assume that estimates are needed and provide value based on their accuracy.
I’ve been reading (really, listening to) Daniel Kahneman’s “Thinking, Fast and Slow“, and I’m continually amazed at the valuable information in it. The chapter I’m currently listening to discusses the Planning Fallacy, which is how most people don’t pay attention to how similar projects work out when examining your own project. For example (from the book), you’re planning to create a new educational curriculum and, after a year, you estimate that it will take another 2 years to complete. You then look at other projects that are similar (also creating a curriculum of similar scope) and find out that 40% outright fail to finish and the rest took 7-10 years to complete. As Kahneman says, “People who have information about an individual case rarely feel the need to know the statistics of the class to which the case belongs.” In other words, you underestimate how long it will take to finish a task, despite having knowledge that similar tasks have generally taken longer than planned. Sound familiar?
As a way to avoid the Planning Fallacy, he refers to a paper by Bent Flyvbjerg that talks about “Reference Class Forecasting…which achieves accuracy in projections by basing them on actual performance in a reference class of comparable actions and thereby bypassing both optimism bias and strategic misrepresentation”. The paper discusses a database of public works transportation projects (roads, rail, etc.) and how that can be used to adjust the estimates of new projects upwards, depending on if they were underestimated due to optimistic bias, or intentional misrepresentation (low-balling), or both.
So, the question I have, is can we collect information about our software projects to create reference classes to help us adjust our estimates away from the optimistic side? Can we do it in a way that’s not burdensome and provides value? Note that this isn’t suggesting that we simply plug numbers into a formula, but simply that we use the reference class information to adjust our estimates so that they’re less biased. Is it worth doing this just inside the team at the story (or perhaps feature) level?