5D Library › Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis · 中文

Chapter 2 Why Most Published Findings May Be False: Ioannidis’s Derivation

From diagnostic tests to research findings

Whether someone with a positive screening result really has the disease depends not only on the test’s accuracy but on how common the disease is; Chapter 3 of volume 26 discussed this “base rate” problem. In 2005 the epidemiologist John Ioannidis applied the same reasoning to research itself (Ioannidis 2005). Treat all the “relationships” tested in a field (a gene and a disease, an intervention and an outcome, and so on) as the screened population; a “true relationship” plays the part of disease, the statistical test is the screen, and “significant” is a positive result. The probability that a published significant finding is true, the positive predictive value (PPV), then depends on three things: what share of the tested relationships are true, the power of the tests, and the significance level.

The derivation

Let RR be the ratio of true relationships to no relationships among those tested (the pre-study odds), 1−β1 – \beta the power and α\alpha the significance level. Of cc relationships tested, cR/(R+1)cR/(R+1) are true, of which a share (1−β)(1-\beta) is detected; c/(R+1)c/(R+1) are not, of which a share α\alpha is falsely flagged. So

PPV=(1−β)R(1−β)R+α=(1−β)RR−βR+α.\mathrm{PPV} = \frac{(1-\beta)R}{(1-\beta)R + \alpha} = \frac{(1-\beta)R}{R – \beta R + \alpha}.

Ioannidis then introduces bias uu: of the analyses that would not otherwise have produced “findings”, a share uu is nevertheless reported as findings because of bias in design, analysis or reporting. The share of true relationships reported as findings becomes (1−β)+uβ(1-\beta) + u\beta, and of null relationships α+u(1−α)\alpha + u(1-\alpha):

PPV=(1−β)R+uβRR+α−βR+u−uα+uβR.\mathrm{PPV} = \frac{(1-\beta)R + u\beta R}{R + \alpha – \beta R + u – u\alpha + u\beta R}.

This is how the original paper writes it. The author’s framework writes [(1−β)R+uβR]/[(1−β)R+α+uβR+u(1−α)][(1-\beta)R + u\beta R]\,/\,[(1-\beta)R + \alpha + u\beta R + u(1-\alpha)]. Expanding the denominator, (1−β)R=R−βR(1-\beta)R = R – \beta R and u(1−α)=u−uαu(1-\alpha) = u – u\alpha, so the two denominators agree term by term and the forms are identical.

The two forms compared point by point; the framework’s example (power 0.20, R=0.1R = 0.1, u=0.20u = 0.20); the nine rows of Table 4 of Ioannidis’s paper recomputed; and the formula for nn independent teams testing the same question, where a “finding” means at least one significant result: PPV=R(1−βn)/(R+1−(1−α)n−Rβn)\mathrm{PPV} = R(1-\beta^n)/(R + 1 – (1-\alpha)^n – R\beta^n).

Calculation

# Positive predictive value of a "research finding" (Ioannidis 2005, PLoS Med 2: e124). R
# = pre-study odds of a true relationship among those probed, 1 - beta = power, alpha =
# 0.05, u = share of analyses that would not have been findings but are reported as
# findings through bias.
def ppv_ioannidis(power, R, u, alpha=0.05):
    beta = 1 - power
    return (((1 - beta) * R + u * beta * R)
            / (R + alpha - beta * R + u - u * alpha + u * beta * R))

def ppv_author(power, R, u, alpha=0.05):
    # the framework's form: [(1-b)R + u b R] / [(1-b)R + alpha + u b R + u(1-alpha)]
    beta = 1 - power
    num = (1 - beta) * R + u * beta * R
    return num / ((1 - beta) * R + alpha + u * beta * R + u * (1 - alpha))

def ppv_teams(power, R, n, alpha=0.05):
    # n independent teams probe the same question; a "finding" if at least one is
    # significant
    beta = 1 - power
    return R * (1 - beta ** n) / (R + 1 - (1 - alpha) ** n - R * beta ** n)

if __name__ == "__main__":
    grid = [(p / 20, r / 10, u / 20) for p in range(1, 20) for r in range(1, 31)
            for u in range(21)]
    diff = max(abs(ppv_ioannidis(*g) - ppv_author(*g)) for g in grid)
    print(f"largest difference between the two forms over {len(grid)} parameter sets:"
          f" {diff:.1e}")
    vals = ",  ".join(f"{ppv_ioannidis(0.20, 0.1, u):.4f} (u = {u})"
                      for u in (0.2, 0.3, 0.0))
    print("power 0.20, R = 0.1: PPV = " + vals)
    print("Table 4 of Ioannidis 2005, recomputed:  power   R        u     PPV (paper)")
    rows = [(0.80, 1, 0.10, "0.85"), (0.95, 2, 0.30, "0.85"), (0.80, 1 / 3, 0.40, "0.41"),
            (0.20, 1 / 5, 0.20, "0.23"), (0.20, 1 / 5, 0.80, "0.17"),
            (0.80, 1 / 10, 0.30, "0.20"),
            (0.20, 1 / 10, 0.30, "0.12"), (0.20, 1 / 1000, 0.80, "0.0010"),
            (0.20, 1 / 1000, 0.20, "0.0015")]
    for pw, R, u, paper in rows:
        f = ppv_ioannidis(pw, R, u)
        print(f"{'':39s}{pw:.2f}  {R:7.4f}  {u:.2f}  {f:.4f} ({paper})")
    print("n independent teams, no bias, R = 0.1:  n = 1      5      10")
    for pw in (0.20, 0.80):
        vals = "  ".join(f"{ppv_teams(pw, 0.1, n):.3f}" for n in (1, 5, 10))
        print(f"   power {pw:.2f}{'':27s}" + vals)
# largest difference between the two forms over 11970 parameter sets: 1.1e-15
# power 0.20, R = 0.1: PPV = 0.1304 (u = 0.2),  0.1161 (u = 0.3),  0.2857 (u = 0.0)
# Table 4 of Ioannidis 2005, recomputed:  power   R        u     PPV (paper)
#                                        0.80   1.0000  0.10  0.8497 (0.85)
#                                        0.95   2.0000  0.30  0.8521 (0.85)
#                                        0.80   0.3333  0.40  0.4055 (0.41)
#                                        0.20   0.2000  0.20  0.2308 (0.23)
#                                        0.20   0.2000  0.80  0.1718 (0.17)
#                                        0.80   0.1000  0.30  0.2043 (0.20)
#                                        0.20   0.1000  0.30  0.1161 (0.12)
#                                        0.20   0.0010  0.80  0.0010 (0.0010)
#                                        0.20   0.0010  0.20  0.0015 (0.0015)
# n independent teams, no bias, R = 0.1:  n = 1      5      10
#    power 0.20                           0.286  0.229  0.182
#    power 0.80                           0.615  0.306  0.199

Over 11,970 parameter sets the largest difference between the two forms is of order 10−1510^{-15}, mere rounding error. The framework’s example gives 0.1304, about 13.0 %, not the author’s estimate of about 12.8 %; the row of Ioannidis’s Table 4 with power 0.20 and RR of 1:10 reads 0.12, but its bias is u=0.30u = 0.30 (the program gives 0.1161). With no bias at all, the same power and odds give 0.2857. All nine rows of Table 4 are reproduced by the formula.

Figure 1 plots these relationships. With power 0.80 and no bias, the pre-study odds must exceed about 1:16 for the PPV to pass one half; drop the power to 0.20, or add bias u=0.2u = 0.2, and the whole curve shifts right, so that at the odds typical of exploratory research most “findings” are false.

Positive predictive value against pre-study odds R (log axis). Solid lines without bias, dashed with bias u = 0.2; the gold point is the framework’s example (power 0.20, R = 0.1, u = 0.2, PPV = 0.13).

The multi-team formula shows something else: when several teams test the same question and any one significant result gets reported, at power 0.20 the PPV falls from 0.286 for one team to 0.229 for five and 0.182 for ten. Ioannidis drew a set of corollaries: the smaller the studies, the smaller the effects, the more relationships tested and the less they were pre-selected, the more flexible the designs and analyses, the greater the financial and other interests, and the hotter the field, the less likely its findings are to be true (Ioannidis 2005).

Checking the formula by simulation. Each study is a two-sided zz test; true relationships make up R/(1+R)R/(1+R) of those probed (R=0.1R = 0.1); their mean zz gives power 0.20 or 0.80; with bias, a non-significant analysis is still reported as a finding with probability uu. Each case simulates 400,000 studies. The last line of each block groups the significant findings by pp-value and shows the share that are true.

Calculation

# The PPV formula checked by simulation, and Goodman & Greenland's (2007) point that p =
# 0.049 and p = 0.0001 are not equally strong evidence. Two-sided z-tests; true
# relationships are R / (1 + R) of those probed (R = 0.1), with mean z giving power 0.20
# or 0.80; with bias u a non-significant analysis is still reported as a finding with
# probability u.
import math, random
from statistics import NormalDist
N = NormalDist()

def simulate(rng, R, delta, u, n=400000):
    studies = []
    for _ in range(n):
        true = rng.random() < R / (1 + R)
        z = rng.gauss(delta if true else 0.0, 1)
        p = 2 * (1 - N.cdf(abs(z)))
        finding = p < 0.05 or rng.random() < u
        studies.append((true, p, finding))
    return studies

def ppv(rows):
    found = [t for t, p, f in rows if f]
    return sum(found) / len(found)

if __name__ == "__main__":
    rng = random.Random(2005)
    for power, delta in ((0.20, 1.96 - 0.8416), (0.80, 1.96 + 0.8416)):
        for u in (0.0, 0.2):
            s = simulate(rng, 0.1, delta, u)
            beta = 1 - power
            R, a = 0.1, 0.05
            f = (power * R + u * beta * R) / (R + a - beta * R + u - u * a + u * beta * R)
            print(f"power {power:.2f}, u = {u:.1f}: simulated PPV {ppv(s):.3f}"
                  f"   formula {f:.3f}")
        s = simulate(rng, 0.1, delta, 0.0)
        line = []
        for lo, hi in ((0.0, 0.001), (0.001, 0.01), (0.01, 0.03), (0.03, 0.05)):
            b = [t for t, p, f in s if lo <= p < hi]
            line.append(f"{lo:g}-{hi:g}: {sum(b) / len(b):.2f}")
        print("   no bias, share true by p:  " + ";  ".join(line))
# power 0.20, u = 0.0: simulated PPV 0.288   formula 0.286
# power 0.20, u = 0.2: simulated PPV 0.131   formula 0.130
#    no bias, share true by p:  0-0.001: 0.60;  0.001-0.01: 0.39;  0.01-0.03: 0.28;  0.03-0.05: 0.21
# power 0.80, u = 0.0: simulated PPV 0.618   formula 0.615
# power 0.80, u = 0.2: simulated PPV 0.260   formula 0.259
#    no bias, share true by p:  0-0.001: 0.97;  0.001-0.01: 0.76;  0.01-0.03: 0.42;  0.03-0.05: 0.24

Simulation and formula agree: at power 0.20, 0.288 against 0.286 without bias and 0.131 against 0.130 with it; at power 0.80, 0.618 against 0.615 and 0.260 against 0.259. The differences are sampling error.

Criticism one: cutting pp-values in two

In 2007 Goodman and Greenland set out a systematic critique of Ioannidis’s argument (Goodman and Greenland 2007). Their main point was that the model looks only at the verdict “significant or not”, ignoring the actual pp-value. A result with p=0.049p = 0.049 and one with p=0.0001p = 0.0001 are both just “positive” in the model. The last line of the program shows why this matters: at power 0.80 with no bias, 97 % of findings with p<0.001p < 0.001 are true but only 24 % of those with 0.03≤p<0.050.03 \le p < 0.05; at power 0.20 the figures are 60 % and 21 %. The average “share of published findings that are false” hides very large differences in the strength of evidence.

Their other criticisms include these: adding a bias term uu to the Bayesian calculation, even at small values, drastically weakens the evidential effect of any study, so the conclusion follows to a considerable degree from the assumptions; Ioannidis’s argument that “hotter fields have more false findings” shows only that there are more false positives in absolute number, not a higher proportion; and so the claim that the model proves most research false is unfounded. They also acknowledged that the paper is a useful reminder that the probability of a hypothesis depends on much more than the pp-value, and that it offers useful guidance in areas such as avoiding bias.

Note

Ioannidis’s title and his derivation. “Most published research findings are false” is the conclusion of a conditional: if a field’s pre-study odds, power and bias lie in the ranges he assumed, then most findings are false. The formula itself is not in dispute; it is Bayes’s theorem. The dispute is about the real values of RR, power and uu, which are hard to measure directly. That is why the empirical estimate in the next section caused such argument.

Criticism two: an empirical estimate

Evidence

Jager and Leek’s “science-wise false discovery rate”. In 2014 these two statisticians automatically extracted pp-values from the abstracts of 77,430 papers (5,322 of the abstracts reported at least one pp-value) published between 2000 and 2010 in five leading medical journals (The Lancet, JAMA, the New England Journal of Medicine, the BMJ and the American Journal of Epidemiology). They modelled the pp-values below 0.05 as a mixture: those from true nulls uniform between 0 and 0.05, those from real effects following a beta distribution truncated at 0.05, with rounding and “p<p <”-style censored reporting taken into account. They estimated a false discovery rate of about 14 % (standard deviation 1 %), ranging from 11 % to 19 % across journals, far below “most” (Jager and Leek 2014a).

The paper drew a set of discussion pieces in the same issue of Biostatistics, almost all critical (Gelman and O’Rourke 2014; Ioannidis 2014). The main problems raised were these. The pp-values reported in abstracts are themselves selected, and the model assumes that null pp-values are uniform, which fails under pp-hacking (Chapter 3 will show that optional stopping piles significant pp-values up just below 0.05). The model counts every effect that is “not exactly zero” as “true”, yet whether a real but uselessly small effect counts as a “true finding” is itself contested. And pp-values mined from abstracts cannot represent the whole literature. Ioannidis’s discussion piece was titled “Why ‘An estimate of the science-wise false discovery rate and application to the top medical literature’ is false”. The two authors replied in the same issue (Jager and Leek 2014b); the argument kept returning to what “false” should mean and whether pp-values in abstracts can stand for the literature.

Both sides’ numbers rest on assumptions that are hard to test. Ioannidis’s derivation shows which factors lower the reliability of findings; what a field’s false discovery rate actually is is best answered by repeating the studies (Chapter 5).

Chapter summary

Ioannidis applied the Bayesian reasoning of diagnostic tests to scientific findings: the positive predictive value is fixed by the pre-study odds RR, the power and the significance level, and bias uu lowers it further. The framework’s way of writing the formula is equivalent to the original; with power 0.20, R=0.1R = 0.1 and u=0.20u = 0.20 the PPV is about 13.0 %. Goodman and Greenland pointed out that dichotomising pp-values hides differences in the strength of evidence and that “most are false” depends on assumptions; Jager and Leek’s empirical estimate (about 14 %) rests on contested assumptions of its own. The next chapter looks at where bias uu comes from: the researcher’s freedom in analysing data.

References

Gelman, Andrew, and Keith O’Rourke. 2014. “Discussion: Difficulties in Making Inferences about Scientific Truth from Distributions of Published p-Values.” Biostatistics 15 (1): 18–23. https://doi.org/10.1093/biostatistics/kxt034.
Goodman, Steven, and Sander Greenland. 2007. “Why Most Published Research Findings Are False: Problems in the Analysis.” PLoS Medicine 4 (4): e168. https://doi.org/10.1371/journal.pmed.0040168.
Ioannidis, John P. A. 2005. “Why Most Published Research Findings Are False.” PLoS Medicine 2 (8): e124. https://doi.org/10.1371/journal.pmed.0020124.
Ioannidis, John P. A. 2014. “Discussion: Why ‘an Estimate of the Science-Wise False Discovery Rate and Application to the Top Medical Literature’ Is False.” Biostatistics 15 (1): 28–36. https://doi.org/10.1093/biostatistics/kxt036.
Jager, Leah R., and Jeffrey T. Leek. 2014a. “An Estimate of the Science-Wise False Discovery Rate and Application to the Top Medical Literature.” Biostatistics 15 (1): 1–12. https://doi.org/10.1093/biostatistics/kxt007.
Jager, Leah R., and Jeffrey T. Leek. 2014b. “Rejoinder: An Estimate of the Science-Wise False Discovery Rate and Application to the Top Medical Literature.” Biostatistics 15 (1): 39–45. https://doi.org/10.1093/biostatistics/kxt038.

Fan, Y. (2026). Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis: From Popper's Demarcation and p-Hacking to Ioannidis's Theorem and Pre-Registration, Read Beside Classical Chinese Evidential Scholarship [科学方法论、可证伪性与可重复性危机的数理统计] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23782104 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).