Two kinds of failure

Imagine a learner practising a judgement: is there a shadow on this X-ray, did this melody go up or down, is this sentence grammatical? If every item is easy, the learner gets nearly all of them right, and precisely because they are right, each item carries little new information. If every item is very hard, the learner is close to guessing, right and wrong have little to do with their judgement, and again little is learned. Intuition says the most effective difficulty lies somewhere in between.

That intuition has a long history in education and psychology, but it has usually remained qualitative. The paper by Wilson, Shenhav, Straccia and Cohen in Nature Communications in 2019 calculated “in between” within an explicit mathematical model (Wilson et al. 2019).

The model: precision, difficulty and error-driven learning

The paper considers binary classification. On each trial the learner computes a decision variable $h$ from the stimulus and decides according to its sign. $h$ is a noisy version of the true decision variable $\Delta$: $\Delta$ says how far the stimulus lies from the category boundary, and the smaller $|\Delta|$, the harder the item. The noise has scale $1/\beta$, where $\beta$ is the precision, standing for the learner’s skill, the thing learning has to improve. If the standardised noise has distribution function $F$ and density $p$, the error rate is

$$\mathrm{ER}(\beta, \Delta) = F(-\beta \Delta).$$

The learner reduces its error rate by gradient descent on internal parameters $\phi$. By the chain rule,

$$\frac{\partial \mathrm{ER}}{\partial \phi} = \frac{\partial \mathrm{ER}}{\partial \beta}\cdot\frac{\partial \beta}{\partial \phi},$$

and only the first factor depends on the difficulty $\Delta$. So, at a given level of skill, the expected learning gain per trial is greatest when

$$-\frac{\partial \mathrm{ER}}{\partial \beta} = \Delta\, p(\beta\Delta)$$

is greatest. Differentiating with respect to $\Delta$ and setting the result to zero gives $p(x) + x\,p'(x) = 0$ with $x = \beta\Delta$. For Gaussian noise $p'(x) = -x\,p(x)$, so $1 – x^2 = 0$, that is $\beta\Delta^* = 1$: the optimal difficulty makes the “true signal” exactly one noise standard deviation. The error rate there is

$$\mathrm{ER}^* = F(-1) = \Phi(-1) = \tfrac{1}{2}\,\mathrm{erfc}\!\left(\tfrac{1}{\sqrt{2}}\right) \approx 0.1587,$$

an accuracy of about 84.13 %. This is where the “85 % rule” comes from. Note that it does not depend on $\beta$: whatever the learner’s current level, the optimal error rate is the same; what changes is the difficulty needed to produce it.

A brute-force numerical reproduction of the derivation: for three noise distributions, search at fixed precision $\beta = 1$ for the difficulty that maximises $\Delta\,p(\beta\Delta)$; then see how much of the per-trial learning gain is left when training departs from the optimal error rate.

# Optimal training error rate (Wilson et al. 2019): ER = F(-beta Delta), gradient learning
# gain -dER/dbeta = Delta p(beta Delta); find the Delta that maximises it, for three noises.
import math
from statistics import NormalDist
N = NormalDist()

def best(F, p, beta=1.0):
    # brute-force search over difficulty Delta for the largest expected learning gain
    grid = [i / 10000 for i in range(1, 60001)]
    d = max(grid, key=lambda D: D * p(beta * D))
    return d, F(-beta * d), d * p(beta * d)

noises = {
    "Gaussian": (N.cdf, N.pdf),
    "Laplacian": (lambda x: 0.5 * math.exp(x) if x < 0 else 1 - 0.5 * math.exp(-x),
                  lambda x: 0.5 * math.exp(-abs(x))),
    "Cauchy": (lambda x: 0.5 + math.atan(x) / math.pi, lambda x: 1 / (math.pi * (1 + x * x))),
}
print("noise      | optimal beta*Delta | optimal error rate | accuracy")
for name, (F, p) in noises.items():
    d, er, _ = best(F, p)
    print(f"{name:10s} | {d:18.3f} | {er:18.4f} | {1 - er:.2%}")
print(f"closed form, Gaussian: 1/2 erfc(1/sqrt 2) = {0.5 * math.erfc(1 / math.sqrt(2)):.5f}")
wrong = 0.5 * (1 - 1 / math.sqrt(2))
print(f"the formula 1/2 (1 - 1/sqrt 2)       = {wrong:.5f}  (not the result)")

print("\nGaussian noise: learning gain per trial relative to the optimum, beta = 1")
g_star = best(N.cdf, N.pdf)[2]
for er in (0.01, 0.05, 0.10, 0.1587, 0.20, 0.30, 0.40, 0.45):
    D = -N.inv_cdf(er)
    g = D * N.pdf(D) / g_star
    print(f"  training error rate {er:6.2%} (accuracy {1 - er:6.2%}): gain {g:5.1%}")
# noise      | optimal beta*Delta | optimal error rate | accuracy
# Gaussian   |              1.000 |             0.1587 | 84.13%
# Laplacian  |              1.000 |             0.1839 | 81.61%
# Cauchy     |              1.000 |             0.2500 | 75.00%
# closed form, Gaussian: 1/2 erfc(1/sqrt 2) = 0.15866
# the formula 1/2 (1 - 1/sqrt 2)       = 0.14645  (not the result)
#
# Gaussian noise: learning gain per trial relative to the optimum, beta = 1
#   training error rate  1.00% (accuracy 99.00%): gain 25.6%
#   training error rate  5.00% (accuracy 95.00%): gain 70.1%
#   training error rate 10.00% (accuracy 90.00%): gain 92.9%
#   training error rate 15.87% (accuracy 84.13%): gain 100.0%
#   training error rate 20.00% (accuracy 80.00%): gain 97.4%
#   training error rate 30.00% (accuracy 70.00%): gain 75.4%
#   training error rate 40.00% (accuracy 60.00%): gain 40.5%
#   training error rate 45.00% (accuracy 55.00%): gain 20.6%

Integrate the learning equation and compare training that resets the difficulty to the optimal error rate at every step with training whose difficulty stays at its starting value.

# Learning dynamics in the model of Wilson et al. (2019), Gaussian noise. Integrate
#   d beta / dt = eta * Delta * p(beta * Delta)
# two ways: (a) difficulty re-set every step so that the error rate stays at
# ER* = Phi(-1) = 15.87 % (Delta = 1 / beta); (b) difficulty frozen at its starting value.
import math
from statistics import NormalDist
N = NormalDist()

eta, beta0 = 0.01, 1.0
for label, adapt in (("(a) error rate held at 15.87%", True),
                     ("(b) difficulty fixed at start", False)):
    beta, D0, t, out = beta0, 1.0 / beta0, 0, []
    for T in (100, 1000, 10000, 100000):
        while t < T:
            D = 1.0 / beta if adapt else D0
            beta += eta * D * N.pdf(beta * D)
            t += 1
        D = 1.0 / beta if adapt else D0
        out.append(f"t={T:>6}: beta {beta:5.2f}, ER {N.cdf(-beta * D):6.2%}")
    print(label + "\n   " + "\n   ".join(out))
print(f"theory for (a): beta = sqrt(beta0^2 + 2 eta p(1) t) = "
      f"{math.sqrt(beta0 ** 2 + 2 * eta * N.pdf(1) * 100000):.2f} at t = 100000")
# (a) error rate held at 15.87%
#    t=   100: beta  1.22, ER 15.87%
#    t=  1000: beta  2.42, ER 15.87%
#    t= 10000: beta  7.03, ER 15.87%
#    t=100000: beta 22.02, ER 15.87%
# (b) difficulty fixed at start
#    t=   100: beta  1.22, ER 11.21%
#    t=  1000: beta  2.06, ER  1.98%
#    t= 10000: beta  3.06, ER  0.11%
#    t=100000: beta  3.80, ER  0.01%
# theory for (a): beta = sqrt(beta0^2 + 2 eta p(1) t) = 22.02 at t = 100000

The program shows several things. First, for all three noise distributions the optimum lies at $\beta\Delta = 1$, but the corresponding error rates differ: 15.87 % for Gaussian noise, 18.39 % for Laplacian noise, 25 % for Cauchy noise. “85 %” depends on the shape of the noise, as the paper itself states (Wilson et al. 2019). Second, the optimum is quite flat: with training error rates between 10 % and 20 % the per-trial gain stays at or above about 93 % of its maximum (92.9 % at 10 %), and at 30 % it is still about 75 %. The real losses are at the extremes: at an error rate of 1 % (99 % correct) only about 26 % of the gain remains, and at 40 % about 40.5 %. Third, the long-run difference is larger still. When difficulty keeps pace with skill, precision grows as $\sqrt{t}$, reaching 22.02 after 100,000 trials, exactly as the formula predicts; when difficulty is held fixed, the learner soon becomes “too good”, the error rate falls to about 2 % after 1,000 trials, and precision is only 3.80 after 100,000.

Simulating a perceptron learner

The derivation assumes that the trainer knows the learner’s current precision at every step and sets the difficulty accordingly. In a simulation this can be done. The learner below learns an unknown classification direction in 50 dimensions, with unit noise in every dimension; its precision is the cosine of the angle between its weight vector and the true direction.

The learning rule is a smoothed (“logistic”) version of the perceptron rule: the more wrong or unsure, the larger the update. For each target error rate, 60 learners are simulated, each trained for 400 trials; skill is then assessed on a test at fixed difficulty $\Delta = 1$.

# Simulated learner in 50 dimensions: x = label*Delta*e + N(0,1) noise; precision beta =
# cos(w, e). Delta is set before each trial so the CURRENT error rate equals the target.
# Rule: smoothed (logistic) perceptron update. Skill: test accuracy at Delta = 1.
import math, random
from statistics import NormalDist
N = NormalDist()
D, TRIALS, ETA, RUNS = 50, 400, 0.05, 60

def train(target_er, seed, trials=TRIALS, record=()):
    rng = random.Random(seed)
    w = [rng.gauss(0, 1) for _ in range(D)]
    w[0] = abs(w[0]) + 0.5                                  # start slightly better than chance
    curve = []
    for t in range(trials):
        norm = math.sqrt(sum(v * v for v in w))
        beta = w[0] / norm
        if t in record:
            curve.append(N.cdf(beta))                       # test accuracy at Delta = 1
        delta = -N.inv_cdf(target_er) / max(beta, 1e-3)
        lab = 1 if rng.random() < 0.5 else -1
        x = [rng.gauss(0, 1) for _ in range(D)]
        x[0] += lab * delta
        h = sum(a * b for a, b in zip(w, x)) / norm
        g = 1 - 1 / (1 + math.exp(-3 * lab * h))            # large when wrong or unsure
        w = [a + ETA * g * lab * b for a, b in zip(w, x)]
    norm = math.sqrt(sum(v * v for v in w))
    return w[0] / norm, curve

if __name__ == "__main__":
    targets = (0.02, 0.05, 0.10, 0.13, 0.16, 0.20, 0.25, 0.30, 0.40)
    print(f"{RUNS} learners per target, {TRIALS} trials each")
    print("target training error | final precision beta | test accuracy at Delta = 1")
    res = {er: sum(train(er, s)[0] for s in range(RUNS)) / RUNS for er in targets}
    for er, b in res.items():
        print(f"        {er:5.0%}         |        {b:5.3f}         |        {N.cdf(b):6.1%}")
    top = max(res, key=res.get)
    near = [e for e in targets if res[e] >= 0.98 * res[top]]
    print(f"best target {top:.0%}; within 2 % of the best precision: "
          + ", ".join(f"{e:.0%}" for e in near))
# 60 learners per target, 400 trials each
# target training error | final precision beta | test accuracy at Delta = 1
#            2%         |        0.625         |         73.4%
#            5%         |        0.727         |         76.6%
#           10%         |        0.789         |         78.5%
#           13%         |        0.805         |         79.0%
#           16%         |        0.813         |         79.2%
#           20%         |        0.814         |         79.2%
#           25%         |        0.804         |         78.9%
#           30%         |        0.781         |         78.3%
#           40%         |        0.667         |         74.8%
# best target 20%; within 2 % of the best precision: 13%, 16%, 20%, 25%

The simulation broadly agrees with the theory, but not exactly. The best target lies around 16–20 %, and the band within 2 % of the best precision runs from 13 % to 25 %; training at a 2 % error rate leaves clearly lower final precision (0.625 against 0.814), and so does training at 40 % (0.667). The optimum may sit a little above 15.87 % because a finite number of trials, discrete learning steps and the smoothed learning rule all depart from the idealised conditions of the derivation (the difference between 16 % and 20 % is only 0.001, within simulation error). This is exactly the point: the optimum is a band, not a number to the second decimal place.

Assumptions and limits

The result’s power comes from its simplicity, and so do its limits. There are at least five assumptions. First, difficulty can be adjusted continuously and the trainer knows the learner’s current level; in reality this can only be estimated from recent successes and errors, and the adaptive “staircases” of Chapter 8 are an approximation. Second, binary decisions; multi-choice tasks, continuous responses and open-ended production have their own error structures, and whether an optimal error rate exists for them, and what it is, must be derived separately. Third, the shape of the noise: as the program showed, changing it moves the optimum from 15.87 % to 18.39 % or 25 %. Fourth, gradient-descent-type learning rules; an ideal Bayesian learner is unaffected by trial order, and learning by memorising examples or by sudden insight may not satisfy the assumption either. Fifth, it optimises learning speed per trial, ignoring motivation, frustration and fatigue, and retention a week later or transfer; later chapters show that spacing and interleaving trade present performance for long-term retention.

So to say that “people should keep 85 % accuracy whatever they learn” is to over-generalise. A safer statement: in practice where item difficulty can be finely adjusted, right and wrong are clear and feedback is prompt, there are theoretical and simulation-based reasons for keeping accuracy roughly between 80 % and 90 %; accuracy that stays near 100 % or near chance is a signal that difficulty needs adjusting.

The zone of proximal development: an analogy, not the same theorem

Vygotsky defined the “zone of proximal development” as the distance between a child’s actual developmental level, as determined by independent problem solving, and the level of potential development, as determined through problem solving under adult guidance or in collaboration with more capable peers (Vygotsky 1978). The concept is about development and social mediation: what a child can do today with help, they may do alone tomorrow; teaching should run ahead of development.

The 85 % rule and the zone of proximal development are indeed alike: both say the most effective learning happens “just beyond current ability”. In their discussion Wilson and colleagues link their result to Metcalfe and Kornell’s “Region of Proximal Learning” model, which describes how learners allocate study time to items not yet mastered but close to mastery, and note that its intellectual roots go back to Piaget and Vygotsky (Metcalfe and Kornell 2005; Wilson et al. 2019). But the two are not the same theorem. The zone of proximal development specifies no optimal accuracy and is concerned with what can be done with the help of others; the 85 % rule is a mathematical result for a single learner, a single skill and a particular learning rule, with no “other” in it. Putting them side by side is an illuminating analogy.

Chapter summary

Wilson and colleagues proved that for binary-classification learners that learn by gradient descent, under Gaussian noise, the per-trial learning gain is greatest when precision times difficulty equals 1, that is at a training error rate of about 15.87 % (about 84.13 % correct); the correct formula is ½ erfc(1/√2), not ½(1 − 1/√2). The optimum is flat, it moves when the shape of the noise changes, and in simulated learners it also falls within a band. The result rests on assumptions (adjustable difficulty, binary decisions, a particular noise, gradient learning) and cannot be generalised into an “85 % law of all learning”. Vygotsky’s zone of proximal development is a related idea, but not the same theorem. The next chapter turns to “deliberate practice” in research on human expertise: how much of the difference between people can the quantity and quality of practice explain?