The Planning Fallacy is now well established in academic literature and there is even a Planning Fallacy wikipedia page. All the other literature I looked at takes this fallacy as a starting point. What the fallacy says is two-fold:
- Humans systematically underestimate how long it will take to do a task
- Humans are over confident in their own estimates
Curiously the literature does show that although human’s can’t estimate how long a task will take, the estimate they produce do correlate with actual time spent on a task. In other words: any time estimate is likely to be too small but relative to other estimates the estimate is good.
Second, it seems the planning fallacy holds retrospectively. If you are asked to record how long you spend on a task you are quite likely to underestimate it. There seems no reason to believe retrospective estimation is significantly more accurate than future estimation.
Something else that comes out of the research is: psychologists and others who study this stuff still don’t completely understand what the brain is up to, some of the studies contradict each other and there are plenty of subtle differences which influence estimates.
Third, although we don’t like to admit it deadlines play a big role in both estimating and doing. If a deadline is mentioned before an estimate is given people tend to estimate within the deadline. People are also quite good at meeting deadlines (assuming no outside blocks that is), partly this is because estimating then doing is a lot about time management. i.e. managing your time to fit within the estimate.
While deadlines seem like a good way of bring work to a conclusion it doesn’t seem a particularly good idea to base deadlines on estimates. Consider two scenarios:
- Simple estimate becomes a deadline: If we ask people to estimate how long a piece of work will take they will probably underestimate. So if this estimate is then used as a deadline the deadline may well be missed.
- Pessimistic estimate becomes deadline: If people are encouraged, coerced, or scared into giving pessimistic estimates the estimate will be too long. If this estimate is then used as a deadline work is likely to be completed inside the time but there will be "slack" time. The actual time spent on the task may be the same either way but the total elapsed (end-to-end) time will be longer.
I would love to spend more time digging into this subject but I can’t. Anyone else want to?