What does Regression to the Mean and the Cover Jinx conclude, in numbers, about “Splitting Variance into Skill and Luck: Repeated Measures, Intraclass Correlation and Reliability”?

Answer: The correlation between two years is the share of the total variance that is skill, and it can be estimated from repeated measurements by a one-way random-effects analysis of variance; but the estimate is not precise, and 60 stores with 4 years of records still give a wide interval. The Spearman-Brown formula tells us how many periods are needed before a ranking means something, and for measures with low single-period reliability the answer is often dozens. Conversion rates under binomial sampling need hundreds or thousands of orders before skill and luck can be told apart. Skill exists, and sometimes it lasts; the use of a variance decomposition is to turn that sentence into a number that can be computed and tested.

Scope of the numbers: Marked boxes separate published evidence, runnable Python calculations (22 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python only

true: var_skill 0.360 var_luck 1.000 ICC 0.2647
one study: var_skill 0.417 var_luck 1.169 ICC 0.2630
500 repeat studies: ICC 2.5% 0.121 median 0.26997.5% 0.397
reliability of a 4-year average0.5902
pull a 4-year average toward the chain mean by0.4098

Key formula

\[y_{it} = \mu + s_i + e_{it},\]
Show the full program
# Chapter 3. p31: Splitting a chain's spread into skill and luck.
# 60 stores observed for 4 years. score = mu + skill + luck with skill sd
# 0.6 and luck sd 1.0 (true single-year reliability 0.36 / 1.36). One-way
# random-effects ANOVA recovers the two variances from the data alone:
#   var_skill = (MSB - MSW) / n,   var_luck = MSW,
#   ICC = var_skill / (var_skill + var_luck).
# We repeat the 4-year study 500 times to show how unsure the estimate is.
# Fixed seed; parameters illustrative.
# 单因素随机效应方差分析:从重复测量中分离技能方差与运气方差。
import math
import random

rng = random.Random(31)
STORES, YEARS = 60, 4
SK, LU = 0.6, 1.0


def study():
    data = []
    for _ in range(STORES):
        s = rng.gauss(0.0, SK)
        data.append([10.0 + s + rng.gauss(0.0, LU) for _ in range(YEARS)])
    grand = math.fsum(math.fsum(r) for r in data) / (STORES * YEARS)
    means = [math.fsum(r) / YEARS for r in data]
    ssb = YEARS * math.fsum((m - grand) ** 2 for m in means)
    ssw = math.fsum(math.fsum((v - m) ** 2 for v in r)
                    for r, m in zip(data, means))
    msb = ssb / (STORES - 1)
    msw = ssw / (STORES * (YEARS - 1))
    vs = max(0.0, (msb - msw) / YEARS)
    return vs, msw, vs / (vs + msw)


if __name__ == "__main__":
    true_icc = SK ** 2 / (SK ** 2 + LU ** 2)
    vs, vl, icc = study()
    print("true:      var_skill {:.3f}  var_luck {:.3f}  ICC {:.4f}"
          .format(SK ** 2, LU ** 2, true_icc))
    print("one study: var_skill {:.3f}  var_luck {:.3f}  ICC {:.4f}"
          .format(vs, vl, icc))
    iccs = sorted(study()[2] for _ in range(500))
    print("500 repeat studies: ICC 2.5% {:.3f}  median {:.3f}  97.5% {:.3f}"
          .format(iccs[12], iccs[250], iccs[487]))
    four = YEARS * true_icc / (1 + (YEARS - 1) * true_icc)
    print("reliability of a {}-year average: {:.4f}".format(YEARS, four))
    print("pull a 4-year average toward the chain mean by: {:.4f}"
          .format(1.0 - four))

Single source and citation

Fan, Y. (2026). Regression to the Mean and the Cover Jinx: Posterior Calibration of Fortune Reports and Serial Independence in the Tale of the Lost Horse (Version 1.0.0), Chapter 3: Splitting Variance into Skill and Luck: Repeated Measures, Intraclass Correlation and Reliability. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-202 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026202,
  author    = {Fan, Yafu},
  title     = {Regression to the Mean and the Cover Jinx: Posterior Calibration of Fortune Reports and Serial Independence in the Tale of the Lost Horse},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book