The inside view and the outside view

Kahneman often told a story. He and several colleagues were writing a textbook, and after a year they discussed how much longer it would take; everyone’s estimate lay between a year and a half and two and a half years. He then asked one of them, an experienced curriculum expert, how long similar teams usually took from that stage. The expert thought for a while and said that, as far as he knew, about four in ten such teams never finished, and those that did took seven to ten years. The book took eight years (Kahneman and Lovallo 1993).

The two ways of forecasting the story contrasts were first set out systematically in Kahneman and Tversky’s 1979 paper (Kahneman and Tversky 1979). The inside view starts from the project itself: our plan, our abilities, the obstacles we can think of. The outside view first asks: what class does this project belong to, and how are outcomes usually distributed in that class? They called the tendency to underestimate completion times, costs and risks systematically, while overestimating benefits, the planning fallacy, and proposed a corrective procedure: select a reference class, assess the distribution of outcomes in that class, make an intuitive estimate, assess how well one can predict the individual case, and accordingly pull the intuitive estimate back towards the mean of the reference class (Kahneman and Tversky 1979).

Transport infrastructure: almost nine in ten over budget

“Almost every project overruns its time and budget” is a common saying. The actual data are more useful, because they tell us how often and by how much.

Flyvbjerg argues that this systematic bias has two complementary sources: unintentional optimism bias, and deliberate strategic misrepresentation, understating costs and overstating benefits to win approval and funding in competition with other projects (Flyvbjerg, Holm, and Buhl 2002; Flyvbjerg 2006). Reference-class forecasting addresses the first; the second needs institutional measures such as independent review and accountability.

# Synthetic reference class: 2,000 lognormal overrun ratios with the rail mean 44.7 %
# and SD 38.4 points of Flyvbjerg et al. (2002). The lognormal shape is an assumption,
# not their data.
import math, random

rng = random.Random(2002)
MEAN, SD = 1.447, 0.384
S2 = math.log(1 + (SD / MEAN) ** 2)
MU, SIG = math.log(MEAN) - S2 / 2, math.sqrt(S2)

def sample(n=2000):
    return sorted(math.exp(rng.gauss(MU, SIG)) for _ in range(n))

def quantile(xs, q):
    i = q * (len(xs) - 1)
    lo = int(i)
    return xs[lo] + (xs[min(lo + 1, len(xs) - 1)] - xs[lo]) * (i - lo)

if __name__ == "__main__":
    xs = sample()
    print(f"lognormal mu = {MU:.4f}, sigma = {SIG:.4f}; synthetic class mean overrun "
          f"{sum(xs) / len(xs) - 1:.1%}")
    over = sum(x > 1 for x in xs)
    print(f"synthetic projects over budget: {over} of {len(xs)}")
    est = 100.0
    med = est * quantile(xs, 0.5)
    print(f"inside-view budget {est:.0f}; outside-view median outcome {med:.1f}")
    print(" accept risk of overrun | budget needed | uplift")
    for risk in (0.50, 0.30, 0.20, 0.10):
        b = est * quantile(xs, 1 - risk)
        print(f"        {risk:.0%}             |    {b:6.1f}     | {b / est - 1:+.0%}")
# lognormal mu = 0.3355, sigma = 0.2609; synthetic class mean overrun 45.2%
# synthetic projects over budget: 1809 of 2000
# inside-view budget 100; outside-view median outcome 140.0
#  accept risk of overrun | budget needed | uplift
#         50%             |     140.0     | +40%
#         30%             |     162.3     | +62%
#         20%             |     175.5     | +75%
#         10%             |     195.9     | +96%

In this synthetic class, 1,809 of the 2,000 projects overrun, about 90 %, slightly more than the 86 % among all of Flyvbjerg’s 258 projects; the two are close only because the lognormal with this mean and standard deviation happens to give it, not as a test against real data. If the team will accept a 50 % risk of overrun, the budget must be raised by 40 %; for a 20 % risk by 75 %; for a 10 % risk by 96 %. The UK rail example in Flyvbjerg’s paper is a 40 % uplift at 50 % risk and 68 % at 10 % risk (Flyvbjerg 2006). The model agrees at 50 % and differs widely at 10 %, which is exactly the point: uplifts should come from the empirical distribution of a real reference class, not from an assumed distributional shape.

Combining one’s own experience with the history of the class

The outside view does not require ignoring the case altogether. If a team has genuinely done better than its peers, that is useful information; the trouble is that a record of a few projects is full of luck. Statistics has a well-developed tool for this: empirical-Bayes shrinkage, which pulls each team’s own average record towards the overall mean according to how reliable it is; the shorter and noisier the record, the more it is pulled (Efron and Morris 1977). This is the same idea as “pulling back towards the mean of the reference class” in Kahneman and Tversky’s corrective procedure.

# Empirical-Bayes shrinkage of 12 teams' own overrun records towards the reference
# class. theta_i ~ N(40, 10^2); project noise SD 30; n_i = 2..8 past projects; sigma
# known, mu and tau^2 by method of moments; B_i = (s^2/n_i) / (s^2/n_i + tau^2). 5,000
# replications.
import random

rng = random.Random(1975)
MU, TAU, SIG = 40.0, 10.0, 30.0
NS = (2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 8, 8)

def one_rep():
    th = [rng.gauss(MU, TAU) for _ in NS]
    yb = [t + rng.gauss(0, SIG / n ** 0.5) for t, n in zip(th, NS)]
    v = [SIG ** 2 / n for n in NS]
    w = [1 / vi for vi in v]
    mu_hat = sum(wi * y for wi, y in zip(w, yb)) / sum(w)
    k = len(NS)
    tau2 = max(0.0, sum((y - mu_hat) ** 2 for y in yb) / (k - 1) - sum(v) / k)
    shr = [(vi / (vi + tau2)) * mu_hat + (tau2 / (vi + tau2)) * y for vi, y in zip(v, yb)]
    se = lambda est: sum((e - t) ** 2 for e, t in zip(est, th)) / k
    return se(yb), se([mu_hat] * k), se(shr), [vi / (vi + tau2) for vi in v]

if __name__ == "__main__":
    reps = 5000
    tot = [0.0, 0.0, 0.0]
    bsum = [0.0] * len(NS)
    for _ in range(reps):
        a, b, c, B = one_rep()
        tot = [tot[0] + a, tot[1] + b, tot[2] + c]
        bsum = [x + y for x, y in zip(bsum, B)]
    print("mean squared error of each team's estimate (percentage points squared):")
    names = ("own record only", "reference-class mean", "empirical-Bayes shrinkage")
    for name, val in zip(names, tot):
        print(f"  {name:26s} {val / reps:7.1f}")
    print("average weight on the reference class, B_i, by number of past projects:")
    cells = [f"n={n}: {bsum[i] / reps:.2f}" for i, n in enumerate(NS) if i % 2 == 0]
    print("  " + "  ".join(cells))
# mean squared error of each team's estimate (percentage points squared):
#   own record only              234.5
#   reference-class mean         109.3
#   empirical-Bayes shrinkage     94.9
# average weight on the reference class, B_i, by number of past projects:
#   n=2: 0.83  n=3: 0.78  n=4: 0.74  n=5: 0.70  n=6: 0.68  n=8: 0.63

Using only the team’s own record gives a mean squared error of 234.5; using only the reference-class mean brings it down to 109.3; the shrinkage estimate is best, at 94.9. For teams with only 2 projects on record, the shrinkage estimate puts about 83 % of the weight on the reference class; for teams with 8 projects this falls to about 63 %, still more than half. With these parameters the true differences between teams (standard deviation 10 points) are much smaller than the random variation of single projects (30 points), so the outside view should dominate. If teams really differed a great deal, or their records were long, more weight would fall on the teams themselves; the shrinkage estimate makes this trade-off automatically from the data.

Chapter summary

The planning fallacy is the tendency to take the inside view and underestimate time and cost systematically; students took longer to write their theses than even their “worst case” estimates. Almost nine in ten (86 %) of 258 transport projects overran their costs, by an average of 44.7 % for rail, 33.8 % for bridges and tunnels and 20.4 % for roads, owing both to optimism bias and to strategic misrepresentation. Reference-class forecasting first finds the distribution of outcomes for similar projects and then places the case within it; a synthetic model shows that the uplift depends on the risk one will accept, and should come from a real empirical distribution. Empirical-Bayes shrinkage combines a team’s own record with the reference class according to its reliability, and clearly beats relying on one’s own record when records are short and noisy. The next chapter turns from individuals to groups.