Can we use Reference Class Forecasting to help make our software estimates more accurate?

[Oct. 4, 2013 – Update Note: changed link to the Flyvbjerg paper thanks to Dave Timmermann’s comment.]

Note: I do not want to debate the usefulness of estimates in software development projects, as that’s a completely separate discussion. For the sake of this thought experiment, assume that estimates are needed and provide value based on their accuracy.

I’ve been reading (really, listening to) Daniel Kahneman’s “Thinking, Fast and Slow“, and I’m continually amazed at the valuable information in it. The chapter I’m currently listening to discusses the Planning Fallacy, which is how most people don’t pay attention to how similar projects work out when examining your own project. For example (from the book), you’re planning to create a new educational curriculum and, after a year, you estimate that it will take another 2 years to complete. You then look at other projects that are similar (also creating a curriculum of similar scope) and find out that 40% outright fail to finish and the rest took 7-10 years to complete. As Kahneman says, “People who have information about an individual case rarely feel the need to know the statistics of the class to which the case belongs.” In other words, you underestimate how long it will take to finish a task, despite having knowledge that similar tasks have generally taken longer than planned. Sound familiar?

As a way to avoid the Planning Fallacy, he refers to a paper by Bent Flyvbjerg that talks about “Reference Class Forecasting…which achieves accuracy in projections by basing them on actual performance in a reference class of comparable actions and thereby bypassing both optimism bias and strategic misrepresentation”. The paper discusses a database of public works transportation projects (roads, rail, etc.) and how that can be used to adjust the estimates of new projects upwards, depending on if they were underestimated due to optimistic bias, or intentional misrepresentation (low-balling), or both.

So, the question I have, is can we collect information about our software projects to create reference classes to help us adjust our estimates away from the optimistic side? Can we do it in a way that’s not burdensome and provides value? Note that this isn’t suggesting that we simply plug numbers into a formula, but simply that we use the reference class information to adjust our estimates so that they’re less biased. Is it worth doing this just inside the team at the story (or perhaps feature) level?

6 thoughts on “Can we use Reference Class Forecasting to help make our software estimates more accurate?

  1. Being a statistician by training and a software guy by trade with plenty of missed deadlines, I wonder if we could use some multivariate classification scheme. For example, I recently completed a project that required a simple web service, 12 data access classes in my object model and a bunch of oracle (that I neither like, understand, or have experience with). It took me three weeks. I wonder if, after a period of time, having a number of those data points on hand could help us use some kind of classification model. It strikes me that most of my projects have a number of attributes but in general, nothing I do is similar to other projects to make reference classes, as I understand them, helpful. Granted, I have not read the paper yet though.

    • Exactly. That’s what the “reference class” aspect is: classify the different types of work, then use historical data for that type of work to make better forecasts.

      Note that’s I’m not a statistician, but it makes enough sense to me that it should work for software.

Leave a Reply

Your email address will not be published. Required fields are marked *