Where the two sayings come from

The most widely quoted line about failing metrics is “when a measure becomes a target, it ceases to be a good measure”. It is often attributed to the economist Charles Goodhart, but these are not his words.

The two laws describe two sides of one thing. Goodhart stresses that the statistical relationship fails (the indicator no longer predicts what we really care about); Campbell stresses that the process being measured is itself distorted (people change their behaviour to suit the indicator). A famous earlier article in the management literature, Kerr’s 1975 “On the folly of rewarding A, while hoping for B”, collected many similar examples (Kerr 1975).

Why it collapses: a selection model

# Goodhart as selection. 20,000 staff: true baseline s ~ N(0, 1), talent for real work
# a ~ U(0.5, 1.5), talent for gaming k ~ U(0, 2), independent (illustrative). Pay b per
# metric point; with costs t^2/(2a) and g^2/(2k) each picks t = b a and g = b k.
# True value V = s + t; metric M = s + t + g + noise (sd 0.5).
import random, statistics as st
rng = random.Random(5)
staff = [(rng.gauss(0, 1), rng.uniform(0.5, 1.5), rng.uniform(0, 2), rng.gauss(0, 0.5))
         for _ in range(20000)]

def top(xs, q=0.10):
    return set(sorted(range(len(xs)), key=lambda i: -xs[i])[:int(len(xs) * q)])

def world(b):
    V = [s + b * a for s, a, k, e in staff]
    M = [s + b * a + b * k + e for s, a, k, e in staff]
    burned = st.mean(b * b * k / 2 for _, _, k, _ in staff)      # cost spent on gaming
    overlap = len(top(M) & top(V)) / (len(V) // 10)
    return st.mean(V), st.mean(M), st.correlation(M, V), overlap, burned

print("  b  | mean V | mean M | corr(M, V) | top 10 % by M also top by V | gaming cost")
for b in (0.0, 0.5, 1.0, 2.0, 4.0):
    mv, mm, r, ov, gc = world(b)
    print(f" {b:3.1f} | {mv:5.2f}  | {mm:5.2f}  |    {r:.2f}    |"
          f"            {ov:5.1%}            |    {gc:5.2f}")
#   b  | mean V | mean M | corr(M, V) | top 10 % by M also top by V | gaming cost
#  0.0 | -0.01  | -0.01  |    0.89    |            67.4%            |     0.00
#  0.5 |  0.49  |  0.99  |    0.87    |            63.5%            |     0.12
#  1.0 |  0.99  |  1.99  |    0.80    |            55.5%            |     0.50
#  2.0 |  1.99  |  3.99  |    0.67    |            43.3%            |     1.99
#  4.0 |  4.00  |  7.99  |    0.54    |            35.0%            |     7.98

With $b = 0$ nobody games; the correlation between metric and true value is 0.89, and of the people in the top 10 % by the metric, 67.4 % are also in the top 10 % by true value. Raising the bonus has three effects. First, true value does rise: the incentive is not useless. Second, the metric rises twice as fast: with these parameters, half of every gain in the metric is gaming. Third, and this is Goodhart’s point, the metric’s ability to tell people apart collapses. At $b = 4$ the correlation falls to 0.54, and only 35.0 % of the metric’s top 10 % really are in the top 10 %; those being rewarded and promoted are increasingly the skilled gamers. Meanwhile the cost spent on gaming grows like $b^2/2$ (7.98 at $b = 4$), all of it wasted.

Nobody in this model “cheats” to the point of breaking the law, and nobody needs bad intentions. As long as the metric can be pushed up more cheaply than by real improvement, and people differ in their talent for the two, the statistical relationship weakens once pressure is applied.

Multitask principal–agent: why optimal incentives should be “blunted”

Book 16 of this series introduced the 1991 multitask model of Holmström and Milgrom: when a job has several dimensions and only some can be measured, strong incentives on the measurable dimensions pull effort away from the unmeasurable ones, so optimal incentives are often weaker than a single-task model suggests (Holmström and Milgrom 1991). Baker showed in 1992 that the heart of the problem is the misalignment between the metric and the true objective: the less the actions to which the metric responds coincide with the actions to which true value responds, the lower the optimal incentive intensity (Baker 1992). Rather than repeat those models, this section takes up a new question: how audits and spot checks change the trade-off.

# Bonus b per metric point; metric = real work t + gaming g; cost (t^2 + g^2)/2 + s t g,
# s = 0.5. Audit probability q; fine F = 2 per unit of gaming found; audits cost K = 0.4
# per unit of q. Goal: total surplus t - cost - K q (pay and fines are transfers).
# Agent's first-order conditions: b = t + s g and b - q F = g + s t (g >= 0).
S, F, K = 0.5, 2.0, 0.4

def agent(b, q):
    g = (b - q * F - S * b) / (1 - S * S)
    return (b, 0.0) if g <= 0 else ((b - S * (b - q * F)) / (1 - S * S), g)

def surplus(b, q):
    t, g = agent(b, q)
    return t - (t * t + g * g) / 2 - S * t * g - K * q, t, g

bs, qs = [i / 200 for i in range(301)], [i / 200 for i in range(201)]
(w0, t0, g0), b0 = max((surplus(b, 0.0), b) for b in bs)
(w1, t1, g1), b1, q1 = max((surplus(b, q), b, q)
                          for b in bs for q in qs)
print(f"no audit    : b = {b0:.3f}, real work {t0:.3f}, gaming {g0:.3f},"
      f" surplus {w0:.3f}")
print(f"random audit: b = {b1:.3f}, q = {q1:.3f}, real work {t1:.3f}, gaming {g1:.3f},"
      f" surplus {w1:.3f}")
print(f"  audits let the organisation pay {b1 / b0:.1f} times as steeply per point")
# Two gaming routes (each costs g^2/2), budget to audit one route's worth, b = 1.
gaming = lambda q_route, b=1.0: max(0.0, b - q_route * F)
print(f"announced audit of route 1 only: gaming {gaming(1.0):.2f} + {gaming(0.0):.2f}")
print(f"unannounced, each route p = 1/2 : gaming {gaming(0.5):.2f} + {gaming(0.5):.2f}")
# no audit    : b = 0.500, real work 0.333, gaming 0.333, surplus 0.167
# random audit: b = 0.900, q = 0.225, real work 0.900, gaming 0.000, surplus 0.405
#   audits let the organisation pay 1.8 times as steeply per point
# announced audit of route 1 only: gaming 0.00 + 1.00
# unannounced, each route p = 1/2 : gaming 0.00 + 0.00

Without audits, the optimal bonus is only $b = 0.5$: any higher, and the waste from extra gaming outweighs the extra real work; real work and gaming are 0.333 each, and total surplus is 0.167. With random audits, the optimum is $q = 0.225$ and $b = 0.9$: the expected fine $qF = 0.45$ is just enough to make gaming unprofitable, the agent puts all effort into real work (0.900), and total surplus rises to 0.405. The value of auditing lies not only in catching gaming but in allowing the organisation to use steeper incentives, here 1.8 times as steep.

The last two lines compare two audit arrangements, each with a budget sufficient to check only one route to gaming. An announced audit (always checking route 1, never route 2) merely shifts gaming to route 2: 0 on the first, 1.00 on the second. If it is not announced which route will be checked, and each is checked with probability one half, the expected fine on both routes is 1.0, just enough to match the bonus, and gaming on both is suppressed.

Designing metrics that resist gaming

Translated into practice, the models of this chapter give four rules:

1. Find the cheapest route to gaming. For each key metric, ask “what is the cheapest way to push it up a little without creating any real value?” If that way is cheaper than real improvement, the metric has a leak.
2. Use several dimensions. Use a few metrics that check each other (on-time delivery and the defect rate recorded by the receiving side, for example), so that gaming any single route leaves traces on another metric.
3. Spot-check at random. Have people who are not rewarded or penalised on the metric check it in a way not announced in advance; the expected penalty from a check must exceed the gain from gaming.
4. Control the steepness of incentives. Until gaming can be contained, keep incentives somewhat “blunt”; once gaming is contained, raise their intensity step by step.

Chapter summary

Goodhart’s 1975 wording was “any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes”; the popular “when a measure becomes a target” comes from Strathern in 1997; Campbell in 1979 stressed that indicators corrupt the process they measure. A selection model shows that as long as gaming is cheaper than real improvement and people differ in their talent for gaming, the higher the bonus, the worse the metric becomes at telling people apart. The main value of random audits is that they allow steeper incentives; audits that announce which route they check merely shift gaming, while unannounced random audits can suppress several routes at once, though they may add risk and crowd out unmeasured effort. The next chapter reads these modern results beside ancient Chinese debates on reward, punishment, names and reality.