Estimation: what happens when you get it wrong?
Author Geoff Barton Published 11 September 2009
Everyone makes estimates, whether they understand that or not. But despite the importance of estimates in our day-to-day lives, and particularly our business lives, most people don’t actually understand how to estimate. Potentially scary stuff.
As Laura Tingle and Nick Lenagham highlighted in the 29–30 August Weekend Financial Review 'Education Minister Julia Gillard says a $1.7 billion blow-out in the cost of the government’s primary school infrastructure stimulus project didn’t stem from problems in the scheme, but because of underestimated demand … A progress report on the $42 billion stimulus package, of which the primary school spend is a big part, was released on Thursday and showed the government had had to divert $1.5 billion from social housing and environmental measures to plug a hole in the primary schools fund … The report said the primary school scheme had underestimated demand by presuming only a 90 per cent take-up rate and that was made worse when schools were allowed to make multiple-funding applications.'
$1.7 billion is a big error in anyone's language. And it was an error that arose out of a simple, common mistake: failing to collect enough data, which resulted in the failure to analyse the inherent risk involved in the estimation.
People estimate things every day—from how long it will take to shower to the number of calories in a piece of chocolate cake. Most estimates are pretty accurate, and the errors tend to balance out in the long run. But some estimates are very important: do I have enough time to cross the road before the approaching car?
Where an estimate is important we need to understand the positive and negative outcomes associated with errors, and to also understand why we are estimating. Sometimes we are estimating for pure accuracy (how many smarties in a jar at the school fete) but most times we are estimating to provide some commitment ('I will have that done by Tuesday', or 'that will cost about $300'). Most business estimates are commitments, which come with asymmetric risks.
If you think about the many estimates that you make, they are most likely estimates. It will most likely take five minutes to get dressed, and the chances of it more or less time are both about 50%. A commitment is something you are expected to achieve. For example, 'I will definitely be ready in five minutes' is a commitment. It is highly likely that you can achieve your commitment, save for natural disasters and other interruptions.
Asymmetric outcomes mean that the impact of over-committing is not the same as the impact of under-committing or vice versa. When racing a train for a level crossing the outcomes are that you save five minutes by not having to wait for the train to pass or you get hit by a train.
Consider a routine car service. The mechanic says, 'I will have your car ready by 4pm Tuesday.' (Commit.) The negative risks associated with the car being ready by 3pm are virtually zero. The customer is not affected; the car simply waits in the yard for one hour. But the negative risk of the car being ready at 5pm involves an upset customer who may not return their business. The consequences of these errors are not the same. The risks are not mirrored or symmetrical about the estimation point. In such cases the mechanic should recognise that they are making a commitment, and have asymmetric risks, and attempt to ensure that it is highly likely that they will achieve the 4pm time (or quantitatively: say a 98% likelihood of being earlier than 4pm, with 2% chance of being later); and the mechanic should then monitor the work being done to change work allocation and priorities to ensure that the car remains on track to be finished by 4pm. The more important the outcome, the more the mechanic should monitor the progress and be careful in making the estimate. This is expected of any business person.
To return to the government's primary school infrastructure blow-out, we can see that the estimate had a $1.7 billion or more downside error attached for underestimating. While the estimate itself should have been made more carefully, monitoring the project during the early stages of implementation should also have been undertaken. This would likely have revealed a higher-than-expected take-up rate. Having recognised this, the risk of blowout could then have been mitigated. Even if the government could not prevent the blow-out, it could certainly have precluded schools from making multiple funding applications—thereby limiting the scale of the error.
This example is a useful case study of the effects of not understanding the importance of the estimation that was made, and then failing to monitor and control the outcomes of the estimate.