5D Library › Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis · 中文

Chapter 5 Large-Scale Replication: Results and Controversy

From inference to measurement

The models of the previous chapters rest on quantities that are hard to measure directly. In the 2010s several groups took another approach: simply run the published studies again.

Three large projects

Evidence

Psychology (Open Science Collaboration 2015). More than two hundred researchers replicated 100 experimental and correlational studies published in 2008 in three psychology journals, using the original materials where possible and high-powered designs based on the original effect sizes (planned power averaging 0.92, median 0.95). Ninety-seven of the originals had reported significant results; of these 97, the replications were significant for 35, that is 36.1 % (95 % confidence interval 26.6 % to 46.2 %). The mean replication effect size (as a correlation rr) was 0.197, about half the original mean of 0.403; 47 % of original effect sizes lay within the replication’s 95 % confidence interval; 39 % of effects were subjectively rated by the teams as replicated; and if no bias in the originals is assumed, combining original and replication left 68 % significant (Open Science Collaboration 2015).

Evidence

Experimental economics (Camerer et al. 2016, Science). Eighteen laboratory experiments published in the American Economic Review and the Quarterly Journal of Economics in 2011–2014 were replicated, all with analysis plans posted in advance and power of at least 90 % (computed from the original effect size). Eleven (61 %) found a significant effect in the same direction; replication effect sizes averaged 66 % of the originals. By other indicators the replication rate lay between 67 % and 78 % (Camerer et al. 2016).

Evidence

Social-science experiments in Nature and Science (Camerer et al. 2018, Nature Human Behaviour). Twenty-one social-science experiments published in Nature and Science in 2010–2015 were replicated with sample sizes on average about five times the originals. Thirteen (62 %) found a significant effect in the same direction; replication effect sizes averaged about 50 % of the originals, about 75 % among successful replications and close to 0 among failed ones. Prediction markets organised in advance by the researchers judged quite accurately which studies would replicate (Camerer et al. 2018).

The framework lists Camerer’s 2016 economics replication alongside the 2018 social-science replication, and the two need distinguishing: the former appeared in Science, the latter in Nature Human Behaviour. Also, the “36 %” of the Open Science Collaboration has the 97 originally significant studies as its denominator, not 100.

What does “replicated” mean?

“36 % replicated” is often read as “64 % of findings are false”. That skips two steps: even for a perfectly real effect, the “92 % power” computed from an exaggerated original effect (Chapter 4) overstates the replication’s real power; and “not significant” does not mean “zero effect”. So the replication rate blends “how many effects are real” with “how large they are”, and cannot be converted directly into a false discovery rate.

A model of a replication project. Of the effects a field studies, a share π\pi is real (true effect dd) and the rest are zero; original studies have 15 to 50 per group and are published only if significant in the claimed direction; each replication is sized from the original’s reported effect for planned power of 92 %; success means significance in the same direction. All parameters are illustrative. The second part computes, for two studies with no bias at all, the probability that one study’s 95 % confidence interval captures the other’s estimate (used in the section on confidence intervals below): the original’s standard error is 1, the replication has kk times its sample size, and τ\tau is the standard deviation of true effects across sites and populations.

Calculation

# A replication project modelled: a share pi of studied effects is real (true d), the rest
# zero; originals (15-50 per group) are published only if p < .05 in the claimed
# direction; replications are sized for 92 % power at the ORIGINAL estimate (as planned in
# OSC 2015). Illustrative values. Part 2, no bias at all: chance that one study's 95 % CI
# captures the other's estimate (original SE = 1, replication k times larger; tau = SD of
# true effects between settings).
import math, random
from statistics import NormalDist
Z, Phi = NormalDist().inv_cdf, NormalDist().cdf

def project(rng, pi, d, n_orig=(15, 50), attempts=100000):
    succ, orig_sum, rep_sum, k, real = 0, 0.0, 0.0, 0, 0
    for _ in range(attempts):
        true_d = d if rng.random() < pi else 0.0
        n = rng.randint(*n_orig)
        se = math.sqrt(2 / n)
        est = rng.gauss(true_d, se)
        if est / se <= 1.96:
            continue                        # not significant: stays in the drawer
        n_rep = math.ceil(2 * (Z(0.975) + Z(0.92)) ** 2 / est ** 2)
        se_r = math.sqrt(2 / n_rep)
        rep = rng.gauss(true_d, se_r)
        k += 1; real += true_d > 0; succ += rep / se_r > 1.96
        orig_sum += est; rep_sum += rep
    return k, real / k, succ / k, rep_sum / orig_sum

if __name__ == "__main__":
    rng = random.Random(2015)
    print("share real  true d | published: share real | replications significant"
          " | effect ratio")
    for pi, d in ((1.0, 0.5), (1.0, 0.2), (0.5, 0.5), (0.5, 0.3), (0.3, 0.5), (0.0, 0.0)):
        k, real, succ, ratio = project(rng, pi, d)
        print(f"   {pi:4.2f}      {d:4.2f}  | {real:22.2f} | {succ:24.0%}"
              f" | {ratio:12.2f}")
    print("replication size  tau | replication in original CI"
          " | original in replication CI")
    for k in (1, 2.5, 5):
        for tau in (0.0, 0.5):
            sd = math.sqrt(1 + 1 / k + 2 * tau ** 2)
            a, b = 2 * Phi(1.96 / sd) - 1, 2 * Phi(1.96 / math.sqrt(k) / sd) - 1
            print(f"   {k:4.1f} x        {tau:3.1f} | {a:26.1%} | {b:26.1%}")
# share real  true d | published: share real | replications significant | effect ratio
#    1.00      0.50  |                   1.00 |                      70% |         0.72
#    1.00      0.20  |                   1.00 |                      22% |         0.32
#    0.50      0.50  |                   0.95 |                      67% |         0.69
#    0.50      0.30  |                   0.90 |                      37% |         0.43
#    0.30      0.50  |                   0.90 |                      64% |         0.66
#    0.00      0.00  |                   0.00 |                       3% |         0.00
# replication size  tau | replication in original CI | original in replication CI
#     1.0 x        0.0 |                      83.4% |                      83.4%
#     1.0 x        0.5 |                      78.5% |                      78.5%
#     2.5 x        0.0 |                      90.2% |                      70.5%
#     2.5 x        0.5 |                      84.5% |                      63.2%
#     5.0 x        0.0 |                      92.6% |                      57.6%
#     5.0 x        0.5 |                      86.7% |                      49.9%

The results show how hard it is to read a replication rate as a false discovery rate. If every effect is real with true d=0.5d = 0.5, the replication rate is 70 % and replication effects average about 72 % of the originals; if every effect is real but true d=0.2d = 0.2, the replication rate is only 22 % and the effect ratio 32 %. In other words, a field with no false findings at all can show a lower replication rate than the Open Science Collaboration did, provided its effects are small and its original studies small. Conversely, when half the effects are zero and real effects are 0.3, 90 % of the published originals are real, yet the replication rate is only 37 % and the effect ratio 0.43, quite close to the Open Science Collaboration’s figures. If every effect is zero, the replication rate is only 3 %.

This does not make a low replication rate “fine”: whether the cause is false findings or exaggerated effects, readers cannot trust those findings at the effect sizes the originals reported.

The controversy: Gilbert et al. and the reply

In 2016 Gilbert, King, Pettigrew and Wilson published a comment in Science arguing that the Open Science Collaboration’s data were “consistent with the opposite conclusion”: that the reproducibility of psychology was in fact quite high (Gilbert et al. 2016). They made three arguments.

  • Error. The Open Science Collaboration’s benchmarks allowed only for sampling error, yet many replications differed from the originals in population and procedure. Drawing on data from the “Many Labs” project (Klein et al. 2014), they argued that with such differences more replications should “fail” by chance alone.
  • Power. In Many Labs, 36 laboratories each replicated every original study, and the project reported that about 85 % of the originals replicated; recomputed with the Open Science Collaboration’s method of a single replication per study, the rate was only about 34 %, suggesting that single replications underestimate reproducibility.
  • Bias. Only 69 % of original authors endorsed the replication protocols; endorsed protocols replicated at 59.7 %, unendorsed ones at only 15.4 %, from which they concluded that departures from the originals biased replications towards failure.

The Open Science Collaboration’s authors replied at once (Anderson et al. 2016): the Many Labs project Gilbert et al. cited was not a representative sample of psychological research, and another multi-site project had a success rate of only about 30 %; by the confidence-interval measure, even without bias the expected replication rate would be only about 78.5 %, against an actual 47.4 %; the link between endorsement and success is correlational, and could arise because original authors less confident in their own studies were less willing to endorse; and 83 % of replication effects were smaller than the originals, as publication bias would predict.

Judging replication by confidence intervals

The dispute kept returning to criteria such as “does the original effect lie within the replication’s confidence interval?” One fact is often overlooked here: even if two studies estimate exactly the same effect without any bias, one study’s 95 % confidence interval does not capture 95 % of the other’s estimates. Cumming and Maillardet showed that when the two studies are equally precise the average is only about 83.4 % (Cumming and Maillardet 2006).

The second part of the earlier program shows that with equally precise studies and no heterogeneity both probabilities are 83.4 %. But the larger the replication, the lower the probability that the original estimate falls within the replication’s interval (the Open Science Collaboration’s measure): 70.5 % when the replication is 2.5 times the original, 57.6 % at 5 times, because a large study’s interval is narrow; heterogeneity between sites (τ=0.5\tau = 0.5) lowers these to 63.2 % and 49.9 %; and if the originals were also filtered for significance, the benchmark is lower still. So the Open Science Collaboration’s 47 % of original effects within the replication interval cannot be compared with “95 %”; a reasonable benchmark depends on the ratio of sample sizes, on heterogeneity and on how strongly the originals were selected. The numbers on both sides of the dispute (such as the expected 78.5 %) also depend on these assumptions.

Note

How this book reads the dispute. Both sides have a point: “failed to replicate” does not mean “the original was false”, yet no reasonable benchmark makes 36 % reassuring. Sturdier than any single replication rate is this: original effect sizes are generally too large, and small originals with pp close to 0.05 are harder to replicate (the Open Science Collaboration also found that stronger original evidence predicted success).

Hypothesis · not established

Not yet established: pre-registered originals replicate more readily. If the researcher degrees of freedom of Chapter 3 are an important cause of failed replication, then among original studies of similar sample size and original pp-value, those with a pre-registered analysis plan should replicate independently more often. If a sufficiently large sample shows no difference, the hypothesis is false.

Chapter summary

Three large replication projects found same-direction significance rates of 36.1 % (psychology, 35/97), 61 % (economics, 11/18) and 62 % (social-science experiments in Nature and Science, 13/21), with replication effect sizes about a half to two-thirds of the originals. A replication rate cannot be converted directly into a false discovery rate: a field with no false findings but small effects can also replicate poorly. Gilbert et al.’s criticisms about heterogeneity, single replications and protocol departures, and the Open Science Collaboration’s reply, each have merit. When replication is judged by confidence intervals, the reasonable benchmark is far below 95 %. The next chapter discusses how to tackle these problems at their source.

References

Anderson, Christopher J. et al. 2016. “Response to Comment on ‘Estimating the Reproducibility of Psychological Science’.” Science 351 (6277): 1037. https://doi.org/10.1126/science.aad9163.
Camerer, Colin F., Anna Dreber, Eskil Forsell, et al. 2016. “Evaluating Replicability of Laboratory Experiments in Economics.” Science 351 (6280): 1433–36. https://doi.org/10.1126/science.aaf0918.
Camerer, Colin F., Anna Dreber, Felix Holzmeister, et al. 2018. “Evaluating the Replicability of Social Science Experiments in Nature and Science Between 2010 and 2015.” Nature Human Behaviour 2 (9): 637–44. https://doi.org/10.1038/s41562-018-0399-z.
Cumming, Geoff, and Robert Maillardet. 2006. “Confidence Intervals and Replication: Where Will the Next Mean Fall?” Psychological Methods 11 (3): 217–27.
Gilbert, Daniel T., Gary King, Stephen Pettigrew, and Timothy D. Wilson. 2016. “Comment on ‘Estimating the Reproducibility of Psychological Science’.” Science 351 (6277): 1037. https://doi.org/10.1126/science.aad7243.
Klein, Richard A. et al. 2014. “Investigating Variation in Replicability: A ‘Many Labs’ Replication Project.” Social Psychology 45 (3): 142–52.
Open Science Collaboration. 2015. “Estimating the Reproducibility of Psychological Science.” Science 349 (6251): aac4716. https://doi.org/10.1126/science.aac4716.

Fan, Y. (2026). Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis: From Popper's Demarcation and p-Hacking to Ioannidis's Theorem and Pre-Registration, Read Beside Classical Chinese Evidential Scholarship [科学方法论、可证伪性与可重复性危机的数理统计] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23782104 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).