5D Library › Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis · 中文
Chapter 2 Why Most Published Findings May Be False: Ioannidis’s Derivation
From diagnostic tests to research findings
Whether someone with a positive screening result really has the disease depends not only on the test’s accuracy but on how common the disease is; Chapter 3 of volume 26 discussed this “base rate” problem. In 2005 the epidemiologist John Ioannidis applied the same reasoning to research itself (Ioannidis 2005). Treat all the “relationships” tested in a field (a gene and a disease, an intervention and an outcome, and so on) as the screened population; a “true relationship” plays the part of disease, the statistical test is the screen, and “significant” is a positive result. The probability that a published significant finding is true, the positive predictive value (PPV), then depends on three things: what share of the tested relationships are true, the power of the tests, and the significance level.
The derivation
Let be the ratio of true relationships to no relationships among those tested (the pre-study odds), the power and the significance level. Of relationships tested, are true, of which a share is detected; are not, of which a share is falsely flagged. So
Ioannidis then introduces bias : of the analyses that would not otherwise have produced “findings”, a share is nevertheless reported as findings because of bias in design, analysis or reporting. The share of true relationships reported as findings becomes , and of null relationships :
This is how the original paper writes it. The author’s framework writes . Expanding the denominator, and , so the two denominators agree term by term and the forms are identical.
The two forms compared point by point; the framework’s example (power 0.20, , ); the nine rows of Table 4 of Ioannidis’s paper recomputed; and the formula for independent teams testing the same question, where a “finding” means at least one significant result: .
Calculation
# Positive predictive value of a "research finding" (Ioannidis 2005, PLoS Med 2: e124). R
# = pre-study odds of a true relationship among those probed, 1 - beta = power, alpha =
# 0.05, u = share of analyses that would not have been findings but are reported as
# findings through bias.
def ppv_ioannidis(power, R, u, alpha=0.05):
beta = 1 - power
return (((1 - beta) * R + u * beta * R)
/ (R + alpha - beta * R + u - u * alpha + u * beta * R))
def ppv_author(power, R, u, alpha=0.05):
# the framework's form: [(1-b)R + u b R] / [(1-b)R + alpha + u b R + u(1-alpha)]
beta = 1 - power
num = (1 - beta) * R + u * beta * R
return num / ((1 - beta) * R + alpha + u * beta * R + u * (1 - alpha))
def ppv_teams(power, R, n, alpha=0.05):
# n independent teams probe the same question; a "finding" if at least one is
# significant
beta = 1 - power
return R * (1 - beta ** n) / (R + 1 - (1 - alpha) ** n - R * beta ** n)
if __name__ == "__main__":
grid = [(p / 20, r / 10, u / 20) for p in range(1, 20) for r in range(1, 31)
for u in range(21)]
diff = max(abs(ppv_ioannidis(*g) - ppv_author(*g)) for g in grid)
print(f"largest difference between the two forms over {len(grid)} parameter sets:"
f" {diff:.1e}")
vals = ", ".join(f"{ppv_ioannidis(0.20, 0.1, u):.4f} (u = {u})"
for u in (0.2, 0.3, 0.0))
print("power 0.20, R = 0.1: PPV = " + vals)
print("Table 4 of Ioannidis 2005, recomputed: power R u PPV (paper)")
rows = [(0.80, 1, 0.10, "0.85"), (0.95, 2, 0.30, "0.85"), (0.80, 1 / 3, 0.40, "0.41"),
(0.20, 1 / 5, 0.20, "0.23"), (0.20, 1 / 5, 0.80, "0.17"),
(0.80, 1 / 10, 0.30, "0.20"),
(0.20, 1 / 10, 0.30, "0.12"), (0.20, 1 / 1000, 0.80, "0.0010"),
(0.20, 1 / 1000, 0.20, "0.0015")]
for pw, R, u, paper in rows:
f = ppv_ioannidis(pw, R, u)
print(f"{'':39s}{pw:.2f} {R:7.4f} {u:.2f} {f:.4f} ({paper})")
print("n independent teams, no bias, R = 0.1: n = 1 5 10")
for pw in (0.20, 0.80):
vals = " ".join(f"{ppv_teams(pw, 0.1, n):.3f}" for n in (1, 5, 10))
print(f" power {pw:.2f}{'':27s}" + vals)
# largest difference between the two forms over 11970 parameter sets: 1.1e-15
# power 0.20, R = 0.1: PPV = 0.1304 (u = 0.2), 0.1161 (u = 0.3), 0.2857 (u = 0.0)
# Table 4 of Ioannidis 2005, recomputed: power R u PPV (paper)
# 0.80 1.0000 0.10 0.8497 (0.85)
# 0.95 2.0000 0.30 0.8521 (0.85)
# 0.80 0.3333 0.40 0.4055 (0.41)
# 0.20 0.2000 0.20 0.2308 (0.23)
# 0.20 0.2000 0.80 0.1718 (0.17)
# 0.80 0.1000 0.30 0.2043 (0.20)
# 0.20 0.1000 0.30 0.1161 (0.12)
# 0.20 0.0010 0.80 0.0010 (0.0010)
# 0.20 0.0010 0.20 0.0015 (0.0015)
# n independent teams, no bias, R = 0.1: n = 1 5 10
# power 0.20 0.286 0.229 0.182
# power 0.80 0.615 0.306 0.199Over 11,970 parameter sets the largest difference between the two forms is of order , mere rounding error. The framework’s example gives 0.1304, about 13.0 %, not the author’s estimate of about 12.8 %; the row of Ioannidis’s Table 4 with power 0.20 and of 1:10 reads 0.12, but its bias is (the program gives 0.1161). With no bias at all, the same power and odds give 0.2857. All nine rows of Table 4 are reproduced by the formula.
Figure 1 plots these relationships. With power 0.80 and no bias, the pre-study odds must exceed about 1:16 for the PPV to pass one half; drop the power to 0.20, or add bias , and the whole curve shifts right, so that at the odds typical of exploratory research most “findings” are false.
The multi-team formula shows something else: when several teams test the same question and any one significant result gets reported, at power 0.20 the PPV falls from 0.286 for one team to 0.229 for five and 0.182 for ten. Ioannidis drew a set of corollaries: the smaller the studies, the smaller the effects, the more relationships tested and the less they were pre-selected, the more flexible the designs and analyses, the greater the financial and other interests, and the hotter the field, the less likely its findings are to be true (Ioannidis 2005).
Checking the formula by simulation. Each study is a two-sided test; true relationships make up of those probed (); their mean gives power 0.20 or 0.80; with bias, a non-significant analysis is still reported as a finding with probability . Each case simulates 400,000 studies. The last line of each block groups the significant findings by -value and shows the share that are true.
Calculation
# The PPV formula checked by simulation, and Goodman & Greenland's (2007) point that p =
# 0.049 and p = 0.0001 are not equally strong evidence. Two-sided z-tests; true
# relationships are R / (1 + R) of those probed (R = 0.1), with mean z giving power 0.20
# or 0.80; with bias u a non-significant analysis is still reported as a finding with
# probability u.
import math, random
from statistics import NormalDist
N = NormalDist()
def simulate(rng, R, delta, u, n=400000):
studies = []
for _ in range(n):
true = rng.random() < R / (1 + R)
z = rng.gauss(delta if true else 0.0, 1)
p = 2 * (1 - N.cdf(abs(z)))
finding = p < 0.05 or rng.random() < u
studies.append((true, p, finding))
return studies
def ppv(rows):
found = [t for t, p, f in rows if f]
return sum(found) / len(found)
if __name__ == "__main__":
rng = random.Random(2005)
for power, delta in ((0.20, 1.96 - 0.8416), (0.80, 1.96 + 0.8416)):
for u in (0.0, 0.2):
s = simulate(rng, 0.1, delta, u)
beta = 1 - power
R, a = 0.1, 0.05
f = (power * R + u * beta * R) / (R + a - beta * R + u - u * a + u * beta * R)
print(f"power {power:.2f}, u = {u:.1f}: simulated PPV {ppv(s):.3f}"
f" formula {f:.3f}")
s = simulate(rng, 0.1, delta, 0.0)
line = []
for lo, hi in ((0.0, 0.001), (0.001, 0.01), (0.01, 0.03), (0.03, 0.05)):
b = [t for t, p, f in s if lo <= p < hi]
line.append(f"{lo:g}-{hi:g}: {sum(b) / len(b):.2f}")
print(" no bias, share true by p: " + "; ".join(line))
# power 0.20, u = 0.0: simulated PPV 0.288 formula 0.286
# power 0.20, u = 0.2: simulated PPV 0.131 formula 0.130
# no bias, share true by p: 0-0.001: 0.60; 0.001-0.01: 0.39; 0.01-0.03: 0.28; 0.03-0.05: 0.21
# power 0.80, u = 0.0: simulated PPV 0.618 formula 0.615
# power 0.80, u = 0.2: simulated PPV 0.260 formula 0.259
# no bias, share true by p: 0-0.001: 0.97; 0.001-0.01: 0.76; 0.01-0.03: 0.42; 0.03-0.05: 0.24Simulation and formula agree: at power 0.20, 0.288 against 0.286 without bias and 0.131 against 0.130 with it; at power 0.80, 0.618 against 0.615 and 0.260 against 0.259. The differences are sampling error.
Criticism one: cutting -values in two
In 2007 Goodman and Greenland set out a systematic critique of Ioannidis’s argument (Goodman and Greenland 2007). Their main point was that the model looks only at the verdict “significant or not”, ignoring the actual -value. A result with and one with are both just “positive” in the model. The last line of the program shows why this matters: at power 0.80 with no bias, 97 % of findings with are true but only 24 % of those with ; at power 0.20 the figures are 60 % and 21 %. The average “share of published findings that are false” hides very large differences in the strength of evidence.
Their other criticisms include these: adding a bias term to the Bayesian calculation, even at small values, drastically weakens the evidential effect of any study, so the conclusion follows to a considerable degree from the assumptions; Ioannidis’s argument that “hotter fields have more false findings” shows only that there are more false positives in absolute number, not a higher proportion; and so the claim that the model proves most research false is unfounded. They also acknowledged that the paper is a useful reminder that the probability of a hypothesis depends on much more than the -value, and that it offers useful guidance in areas such as avoiding bias.
Note
Ioannidis’s title and his derivation. “Most published research findings are false” is the conclusion of a conditional: if a field’s pre-study odds, power and bias lie in the ranges he assumed, then most findings are false. The formula itself is not in dispute; it is Bayes’s theorem. The dispute is about the real values of , power and , which are hard to measure directly. That is why the empirical estimate in the next section caused such argument.
Criticism two: an empirical estimate
Evidence
Jager and Leek’s “science-wise false discovery rate”. In 2014 these two statisticians automatically extracted -values from the abstracts of 77,430 papers (5,322 of the abstracts reported at least one -value) published between 2000 and 2010 in five leading medical journals (The Lancet, JAMA, the New England Journal of Medicine, the BMJ and the American Journal of Epidemiology). They modelled the -values below 0.05 as a mixture: those from true nulls uniform between 0 and 0.05, those from real effects following a beta distribution truncated at 0.05, with rounding and “”-style censored reporting taken into account. They estimated a false discovery rate of about 14 % (standard deviation 1 %), ranging from 11 % to 19 % across journals, far below “most” (Jager and Leek 2014a).
The paper drew a set of discussion pieces in the same issue of Biostatistics, almost all critical (Gelman and O’Rourke 2014; Ioannidis 2014). The main problems raised were these. The -values reported in abstracts are themselves selected, and the model assumes that null -values are uniform, which fails under -hacking (Chapter 3 will show that optional stopping piles significant -values up just below 0.05). The model counts every effect that is “not exactly zero” as “true”, yet whether a real but uselessly small effect counts as a “true finding” is itself contested. And -values mined from abstracts cannot represent the whole literature. Ioannidis’s discussion piece was titled “Why ‘An estimate of the science-wise false discovery rate and application to the top medical literature’ is false”. The two authors replied in the same issue (Jager and Leek 2014b); the argument kept returning to what “false” should mean and whether -values in abstracts can stand for the literature.
Both sides’ numbers rest on assumptions that are hard to test. Ioannidis’s derivation shows which factors lower the reliability of findings; what a field’s false discovery rate actually is is best answered by repeating the studies (Chapter 5).
Chapter summary
Ioannidis applied the Bayesian reasoning of diagnostic tests to scientific findings: the positive predictive value is fixed by the pre-study odds , the power and the significance level, and bias lowers it further. The framework’s way of writing the formula is equivalent to the original; with power 0.20, and the PPV is about 13.0 %. Goodman and Greenland pointed out that dichotomising -values hides differences in the strength of evidence and that “most are false” depends on assumptions; Jager and Leek’s empirical estimate (about 14 %) rests on contested assumptions of its own. The next chapter looks at where bias comes from: the researcher’s freedom in analysing data.
References
Fan, Y. (2026). Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis: From Popper's Demarcation and p-Hacking to Ioannidis's Theorem and Pre-Registration, Read Beside Classical Chinese Evidential Scholarship [科学方法论、可证伪性与可重复性危机的数理统计] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23782104 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).