5D Library › Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis · 中文
Chapter 1 Falsifiability and Research Programmes: What Counts as a Scientific Statement?
A theory that explains everything
Around 1919 the young Karl Popper, in Vienna, met several theories that were then very influential: Einstein’s relativity, Marx’s theory of history, Freud’s psychoanalysis and Adler’s individual psychology. He later recalled that what the followers of the last three said most often was: “Look, the evidence is everywhere.” He gave an example: one man pushes a child into the water intending to drown it, another sacrifices his life trying to save the child. Adler’s theory could say that both were overcoming feelings of inferiority (one needed to prove he dared to commit a crime, the other that he dared to rescue), and Freud’s theory could explain both just as easily (one was repressing, the other sublimating). What troubled Popper was exactly this: whatever happened, the theory was right. Einstein’s theory, by contrast, predicted that light passing near the sun would bend by a definite angle, and the 1919 eclipse observations could have come out otherwise; the theory ran the risk of refutation, and survived the test (Popper 1963).
From this Popper drew his criterion of demarcation: a theory belongs to empirical science if and only if it is falsifiable, that is, if there are conceivable observations that would contradict it (Popper 1959). Three points must be kept apart. First, falsifiability divides empirical science from non-empirical thought; it is not a criterion of meaning, and Popper never held metaphysics to be nonsense. Second, falsifiable is not falsified: a good theory can be refuted and has not yet been. Third, it rests on a logical asymmetry: no number of white swans proves that “all swans are white”, whereas one reliably observed black swan refutes it.
The more it forbids, the more it says
Popper went further: the empirical content of a theory consists of the states of affairs it forbids, the class of its “potential falsifiers”. The more it forbids, the easier it is to refute and the more it says; conversely, the lower its logical probability (Popper 1959). Bar-Hillel and Carnap later turned this idea into a theory of semantic information (Carnap and Bar-Hillel 1952; Bar-Hillel and Carnap 1953). If all possible worlds are equally likely beforehand, and is the proportion of worlds that statement permits (its logical probability), then
A statement true whatever happens (a tautology) permits every world, , and both measures are zero. A statement that forbids every world but one carries the most information.
Note
Semantic information is not Shannon information. The framework attributes this idea to “Carnap/Shannon information theory”, and the two need to be distinguished. Shannon’s 1948 theory of communication concerns the statistical uncertainty of signals, and he states at the outset that the semantic aspects of communication are irrelevant to the engineering problem (Shannon 1948). Bar-Hillel and Carnap borrowed the logarithmic form but replaced statistical probability with logical probability, measuring how many possibilities a sentence excludes. The forms are similar, the concepts different. Their theory also has a famous odd consequence: a self-contradiction excludes every world and so carries maximal information; the authors accepted this, writing that a self-contradictory sentence “asserts too much; it is too informative to be true”.
A toy world: one experiment makes 10 yes/no observations, giving possible worlds. Five theories permit different numbers of worlds. If a theory survives a test (the observed world is one it permits), its likelihood ratio against T0, “anything can happen”, is . Then the real world (each observation “yes” with probability 0.95) repeats the same experiment five times.
Calculation
# Empirical content on a toy world of 2**10 = 1024 equally likely sequences of 10 yes/no
# observations. A theory = the set of worlds it PERMITS; m(h) = permitted / 1024,
# cont(h) = 1 - m(h), inf(h) = -log2 m(h) bits; a survivor's likelihood ratio against
# "anything can happen" is 1 / m(h).
import math, random
from itertools import product
WORLDS = list(product((0, 1), repeat=10))
THEORIES = [
("T0 whatever happens, it fits", lambda w: True),
("T1 at least one 'yes'", lambda w: sum(w) >= 1),
("T2 more 'yes' than 'no'", lambda w: sum(w) > 5),
("T3 at least 9 of 10 'yes'", lambda w: sum(w) >= 9),
("T4 all 10 'yes'", lambda w: sum(w) == 10),
]
def measures(rule):
allowed = sum(1 for w in WORLDS if rule(w))
m = allowed / len(WORLDS)
return allowed, len(WORLDS) - allowed, 1 - m, math.log2(1 / m)
if __name__ == "__main__":
print("theory permits falsifiers cont(h) inf(h) bits"
" LR if it survives")
for name, rule in THEORIES:
a, f, cont, inf = measures(rule)
print(f"{name:29s} {a:7d} {f:11d} {cont:8.4f} {inf:11.3f} {2 ** inf:12.1f}")
# five replications of the same 10-observation experiment; the world really has P(yes)
# = 0.95
rng = random.Random(33)
runs = [tuple(int(rng.random() < 0.95) for _ in range(10)) for _ in range(5)]
print("five runs, number of 'yes' in each:", [sum(r) for r in runs])
for name, rule in THEORIES:
ok = [rule(r) for r in runs]
if all(ok):
bits = 5 * measures(rule)[3]
print(f"{name:29s} survives all 5; evidence gained vs T0 = {bits:5.1f} bits")
else:
print(f"{name:29s} REFUTED in run {ok.index(False) + 1}")
# theory permits falsifiers cont(h) inf(h) bits LR if it survives
# T0 whatever happens, it fits 1024 0 0.0000 0.000 1.0
# T1 at least one 'yes' 1023 1 0.0010 0.001 1.0
# T2 more 'yes' than 'no' 386 638 0.6230 1.408 2.7
# T3 at least 9 of 10 'yes' 11 1013 0.9893 6.541 93.1
# T4 all 10 'yes' 1 1023 0.9990 10.000 1024.0
# five runs, number of 'yes' in each: [10, 9, 9, 10, 10]
# T0 whatever happens, it fits survives all 5; evidence gained vs T0 = 0.0 bits
# T1 at least one 'yes' survives all 5; evidence gained vs T0 = 0.0 bits
# T2 more 'yes' than 'no' survives all 5; evidence gained vs T0 = 7.0 bits
# T3 at least 9 of 10 'yes' survives all 5; evidence gained vs T0 = 32.7 bits
# T4 all 10 'yes' REFUTED in run 2The output turns Popper’s idea into numbers. T0, “whatever happens, it fits”, permits all 1024 worlds, has no potential falsifiers and zero information, and after surviving five tests has gained zero bits of evidence: it never took a risk, so no result can support it. T1, “at least one yes”, forbids a single world and is almost as empty. T3, “at least 9 of 10 yes”, forbids 1013 worlds and gains about 6.5 bits each time it survives, 32.7 bits over five runs. The boldest, T4, “all 10 yes”, carries the most information (10 bits) but is refuted in the second run (which had only 9 yeses). Bold theories gain the most and fail most easily; this is the “risk” Popper spoke of.
“Bits” here are logarithms of likelihood ratios, a Bayesian reading (Popper himself rejected measuring corroboration by probability), borrowed to make one point: a theory that forbids nothing in advance can learn nothing afterwards.
Why “explaining it away afterwards” fails
More common than a literal tautology is the patch applied after failure: the supplement did not work because “it was not taken for long enough”, the model was wrong because “market sentiment was unusual this time”. Each patch, taken alone, might be true; but if patches always appear after failures and predict nothing beyond the failure they explain, the theory in practice becomes T0.
Evidence
“Weak tests” in psychology. In 1967 the psychologist and philosopher of science Paul Meehl pointed out a methodological paradox. In physics, more precise measurement subjects a theory to a more severe test; in the psychology of his day, the usual practice was to test the null hypothesis of “no difference between groups”, which large samples almost always reject, so that more precise measurement made a theory easier to “support” (Meehl 1967). A theory that predicts only “some difference”, without its direction or size, forbids very little.
The Duhem–Quine thesis
Popper’s picture faces a famous difficulty. In 1906 the physicist and historian of science Pierre Duhem pointed out that a physics experiment can never condemn an isolated hypothesis, only a whole theoretical group, because deriving a testable prediction from a hypothesis always requires theories of the instruments, initial conditions and other auxiliary assumptions (Duhem 1954). The philosopher Quine pushed the point further: our statements about the external world face the tribunal of experience not one by one but as a corporate body (Quine 1951). So when a prediction fails, logic tells us only that “the theory and the auxiliary assumptions cannot all be true”, not which to give up.
This does not mean that anything goes. In practice scientists apportion blame according to the evidence each part already has. The philosopher of science Jon Dorling reanalysed the problem with Bayes’s theorem in 1979 (Dorling 1979): how hard a failed prediction hits theory and auxiliary depends on how credible each was beforehand, and on how likely such a failure would be in each of the other cases.
Prediction follows from “theory and auxiliary together”, and fails. and are independent beforehand. The numbers in all three cases are illustrative. The second part writes “patches” as odds: each new auxiliary halves the prior odds; a degenerating patch explains only the failure it was made for (the rival explains it too: likelihood ratio 1), while a progressive patch also predicts a new fact that is then observed, to which the rival gave only a 10 % chance (likelihood ratio 10).
Calculation
# Duhem's problem in Bayesian form (the approach of Dorling 1979; the numbers are
# illustrative). A prediction E follows only from theory T TOGETHER WITH an auxiliary
# assumption A. E fails. Logic says only "not (T and A)"; how the blame is shared depends
# on the priors and on how likely the failure would be in each of the other three cases. T
# and A are independent a priori.
def after_failure(pT, pA, l_TnA, l_nTA, l_nTnA):
# l_xy = P(failure | x, y); P(failure | T, A) = 0 because T and A together entail E
TnA = pT * (1 - pA) * l_TnA
nTA = (1 - pT) * pA * l_nTA
nTnA = (1 - pT) * (1 - pA) * l_nTnA
z = TnA + nTA + nTnA
return TnA / z, nTA / z # P(T | failure), P(A | failure)
cases = [
("well-tested theory, shaky auxiliary", 0.90, 0.60, 0.50, 0.50, 0.50),
("young theory, well-tested auxiliary", 0.50, 0.95, 0.50, 0.50, 0.50),
("old theory, but failure unlikely if T holds", 0.90, 0.60, 0.01, 0.50, 0.50),
]
print("case P(T) before -> after"
" P(A) before -> after")
for label, pT, pA, *l in cases:
T, A = after_failure(pT, pA, *l)
print(f"{label:43s} {pT:.2f} -> {T:.3f} {pA:.2f} -> {A:.3f}")
# Repeated rescue, as odds of the rescued system against a rival (prior odds 9 : 1). Every
# new auxiliary costs a factor 0.5 in prior odds. A DEGENERATING patch only explains the
# failure it was made for (the rival explains it too: likelihood ratio 1). A PROGRESSIVE
# patch also predicts a new fact that is then observed and that the rival gave only a 10 %
# chance (likelihood ratio 10).
print("patches degenerating: P(system) progressive: P(system)")
for k in range(6):
deg, prog = 9 * 0.5 ** k, 9 * (0.5 * 10) ** k
print(f"{k:7d} {deg / (1 + deg):22.3f} {prog / (1 + prog):21.4f}")
# case P(T) before -> after P(A) before -> after
# well-tested theory, shaky auxiliary 0.90 -> 0.783 0.60 -> 0.130
# young theory, well-tested auxiliary 0.50 -> 0.048 0.95 -> 0.905
# old theory, but failure unlikely if T holds 0.90 -> 0.067 0.60 -> 0.560
# patches degenerating: P(system) progressive: P(system)
# 0 0.900 0.9000
# 1 0.818 0.9783
# 2 0.692 0.9956
# 3 0.529 0.9991
# 4 0.360 0.9998
# 5 0.220 1.0000The first case resembles the historical example Dorling discussed: a well-tested theory (0.90 beforehand) with a less reliable auxiliary (0.60). After the failure the theory falls only to 0.783, while the auxiliary drops from 0.60 to 0.130: the blame falls mainly on the auxiliary, as it should. The second case is the reverse: a young theory (0.50) paired with a well-tested auxiliary (0.95) falls to 0.048. The third case shows that if “theory true but auxiliary wrong” would itself rarely produce such a failure, even a theory that was very credible (0.90) falls to 0.067. So the Duhem–Quine thesis describes a logical indeterminacy, but evidence can still discriminate between the parts quite sharply.
Lakatos: progressive and degenerating programmes
In 1970 Imre Lakatos offered a picture closer to the history of science (Lakatos 1970). A research programme has a “hard core” that its researchers agree not to touch, surrounded by an adjustable “protective belt” of auxiliary hypotheses. Adjusting the belt in the face of anomalies is normal; the question is how. If the modified theory predicts new facts, and some of them are confirmed, the programme is progressive; if the modification merely accommodates known anomalies after the fact, bringing no new predictions, or always lags behind the facts, it is degenerating.
The second part of the program shows that after five patches a degenerating programme falls from 0.900 to 0.220: it still “fits” all the data but becomes less and less credible; in a progressive programme each patch’s new prediction is confirmed, and the probability rises to 0.978, 0.996 and on towards 1. The model is the author’s illustration (Lakatos did not use Bayesian language), writing his standard in computable form: judge a patch not by whether it explains known facts, but by whether it predicts unknown ones.
The statistical problems of later chapters are data-analytic versions of “explaining it away afterwards”: choosing the outcome, sample size, covariates or subgroup after seeing the data is like choosing the auxiliary after the prediction has failed. The pre-registration of Chapter 6 asks researchers to write down, before seeing the data, what their theory forbids.
Hypothesis · not established
Not yet established: a testable consequence. If “writing down in advance what is forbidden” really separates progress from degeneration, then within a research field, studies that registered in advance a specific prediction of direction and size should replicate independently more often than studies that did not. If, after controlling for sample size and the strength of the original evidence, the two kinds of study replicate equally often, this consequence is false.
Chapter summary
Popper’s falsifiability defines scientific statements as those that observation could refute; he identified a theory’s empirical content with what it forbids, and Bar-Hillel and Carnap expressed this as semantic information measured by logical probability. A theory that forbids nothing in advance has zero information and cannot be supported by any observation. The Duhem–Quine thesis shows that logic cannot say which part to abandon when a prediction fails, but Bayesian analysis shows that the evidence often apportions blame between theory and auxiliaries quite clearly. Lakatos’s standard is that good modifications predict new facts. The next chapter turns to statistics: if most of the hypotheses tested in a field are false to begin with, how likely is a “significant” finding to be true?
References
Fan, Y. (2026). Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis: From Popper's Demarcation and p-Hacking to Ioannidis's Theorem and Pre-Registration, Read Beside Classical Chinese Evidential Scholarship [科学方法论、可证伪性与可重复性危机的数理统计] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23782104 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).