What does Regression to the Mean and the Cover Jinx conclude, in numbers, about “Splitting Variance into Skill and Luck: Repeated Measures, Intraclass Correlation and Reliability”?
Answer: The correlation between two years is the share of the total variance that is skill, and it can be estimated from repeated measurements by a one-way random-effects analysis of variance; but the estimate is not precise, and 60 stores with 4 years of records still give a wide interval. The Spearman-Brown formula tells us how many periods are needed before a ranking means something, and for measures with low single-period reliability the answer is often dozens. Conversion rates under binomial sampling need hundreds or thousands of orders before skill and luck can be told apart. Skill exists, and sometimes it lasts; the use of a variance decomposition is to turn that sentence into a number that can be computed and tested.
Scope of the numbers: Marked boxes separate published evidence, runnable Python calculations (22 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python only
| true: var_skill 0.360 var_luck 1.000 ICC 0.2647 | |
| one study: var_skill 0.417 var_luck 1.169 ICC 0.2630 | |
| 500 repeat studies: ICC 2.5% 0.121 median 0.269 | 97.5% 0.397 |
| reliability of a 4-year average | 0.5902 |
| pull a 4-year average toward the chain mean by | 0.4098 |
Key formula
Show the full program
# Chapter 3. p31: Splitting a chain's spread into skill and luck.
# 60 stores observed for 4 years. score = mu + skill + luck with skill sd
# 0.6 and luck sd 1.0 (true single-year reliability 0.36 / 1.36). One-way
# random-effects ANOVA recovers the two variances from the data alone:
# var_skill = (MSB - MSW) / n, var_luck = MSW,
# ICC = var_skill / (var_skill + var_luck).
# We repeat the 4-year study 500 times to show how unsure the estimate is.
# Fixed seed; parameters illustrative.
# 单因素随机效应方差分析:从重复测量中分离技能方差与运气方差。
import math
import random
rng = random.Random(31)
STORES, YEARS = 60, 4
SK, LU = 0.6, 1.0
def study():
data = []
for _ in range(STORES):
s = rng.gauss(0.0, SK)
data.append([10.0 + s + rng.gauss(0.0, LU) for _ in range(YEARS)])
grand = math.fsum(math.fsum(r) for r in data) / (STORES * YEARS)
means = [math.fsum(r) / YEARS for r in data]
ssb = YEARS * math.fsum((m - grand) ** 2 for m in means)
ssw = math.fsum(math.fsum((v - m) ** 2 for v in r)
for r, m in zip(data, means))
msb = ssb / (STORES - 1)
msw = ssw / (STORES * (YEARS - 1))
vs = max(0.0, (msb - msw) / YEARS)
return vs, msw, vs / (vs + msw)
if __name__ == "__main__":
true_icc = SK ** 2 / (SK ** 2 + LU ** 2)
vs, vl, icc = study()
print("true: var_skill {:.3f} var_luck {:.3f} ICC {:.4f}"
.format(SK ** 2, LU ** 2, true_icc))
print("one study: var_skill {:.3f} var_luck {:.3f} ICC {:.4f}"
.format(vs, vl, icc))
iccs = sorted(study()[2] for _ in range(500))
print("500 repeat studies: ICC 2.5% {:.3f} median {:.3f} 97.5% {:.3f}"
.format(iccs[12], iccs[250], iccs[487]))
four = YEARS * true_icc / (1 + (YEARS - 1) * true_icc)
print("reliability of a {}-year average: {:.4f}".format(YEARS, four))
print("pull a 4-year average toward the chain mean by: {:.4f}"
.format(1.0 - four))Single source and citation
Fan, Y. (2026). Regression to the Mean and the Cover Jinx: Posterior Calibration of Fortune Reports and Serial Independence in the Tale of the Lost Horse (Version 1.0.0), Chapter 3: Splitting Variance into Skill and Luck: Repeated Measures, Intraclass Correlation and Reliability. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-202 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026202,
author = {Fan, Yafu},
title = {Regression to the Mean and the Cover Jinx: Posterior Calibration of Fortune Reports and Serial Independence in the Tale of the Lost Horse},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · A Conditional Expectation with Slope Below One: Why Top Stores Slip the Next Year
- Chapter 2 · The Flight Instructor's Illusion: Why Criticism Looks More Effective than Praise
- Chapter 4 · Shrinkage Estimation: James and Stein, Efron and Morris's Eighteen Batters, and Shrinkage Ranking
- Chapter 5 · The Cover Jinx: Select at the Peak and You Will See a Fall
- Chapter 6 · Posterior Calibration of Fortune Reports: Buying at the Low Point, Barnum Statements and the Brier Score
- Chapter 7 · Sources East and West: the Huainanzi's Old Man Who Lost His Horse and Galton's 1886 "Regression towards Mediocrity"
- Chapter 8 · Closing the Loop: Annual Reviews by Shrinkage Ranking, Copy without Causal Overstatement, and a Pre-registered Plan