5D Library › Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis · 中文

Preface: Why Believe a Claim?

Note

Please read this first. This book is about scientific methodology and statistics; its programs, models and checklist serve the argument. It is not medical, psychological, investment or legal advice. The checklist in Chapter 8 can help you judge how strong the evidence behind a news story, a business report or a supplement advertisement is, but it cannot replace the advice of a doctor, pharmacist or other professional; do not start, stop or change any treatment or medication on the strength of the checklist alone. The “business report”, “supplement advertisement” and “paper” in the book are fictional examples and refer to no real product, company or author.

A supplement is “clinically proven”; an agency’s case report says its method tripled conversions; a paper’s abstract announces p<0.05p < 0.05. All three claims sound equally confident, yet the evidence behind them may differ enormously. This book asks a plain question: why should we believe a claim?

Answering it takes two sets of tools: the philosophy of science, which says what counts as a scientific statement and why a theory that “can explain whatever happens” says nothing; and statistics, which says how likely a “significant” result is to be true, how much analytic freedom and selective publication inflate false positives and effect sizes, and why locking the analysis in advance makes most of these problems disappear.

What the book covers

  1. Falsifiability and research programmes: Popper’s demarcation criterion and “empirical content”, Bar-Hillel and Carnap’s semantic information, the Duhem–Quine thesis and Lakatos’s research programmes.
  2. Ioannidis’s positive predictive value: the formula checked term by term, with the criticisms of Goodman and Greenland and the empirical estimate of Jager and Leek.
  3. p-hacking and the garden of forking paths: a Monte Carlo re-run of the 2011 simulations of Simmons, Nelson and Simonsohn.
  4. Publication bias and the winner’s curse: the file drawer, funnel-plot tests, and Type M and Type S errors in underpowered studies.
  5. Large-scale replication: what three replication projects did and did not show, and the controversy over replication itself.
  6. The remedies of meta-science: pre-registration, Registered Reports and equivalence tests, with a check of the statistical properties of this series’ decision rule.

Chapter 7 reads these modern debates beside the Book of Han’s “seek truth from facts”, the Doctrine of the Mean’s “without evidence there is no trust”, the Analects’ “the records are insufficient”, and the views on evidence of Ming and Qing textual scholars, and marks the boundary between them. Chapter 8 offers a four-step meta-science checklist for not being fooled, and a pre-registered randomised controlled trial.

This book complements 5D Library volume 26, Probabilistic Judgement, Brier-Score Calibration and the Statistics of Collective Intelligence: that book covers base rates, Bayesian updating and signal detection theory; this one recalls them only briefly where needed, with a reference, and does not repeat its programs.

The book is also the methodological charter of 5D Library: it explains why every book in the series marks published evidence, the author’s calculations and the author’s hypotheses separately, and why every Chapter 8 contains a pre-registered trial. It must therefore apply the same standards to the series itself (Chapters 6 and 8), limitations included.

Claims corrected on checking

Several points in the author’s framework needed correction or qualification once they were checked. Each chapter says so where it arises, and the Editor’s Notes published with the book collect them:

  • Ioannidis’s formula and the worked number. The framework’s form of the positive predictive value is term-by-term identical to the denominator in Ioannidis’s 2005 paper (a program checks nearly twelve thousand parameter sets; the differences are rounding error). But with power 0.20, pre-study odds R=0.1R = 0.1 and bias u=0.20u = 0.20, the PPV is 13.0 %, not about 12.8 %; the “0.12” in Table 4 of the paper corresponds to u=0.30u = 0.30.
  • “Sixty per cent false positives”. The figures for each situation in Table 1 of Simmons et al. (9.5 %, 7.7 %, 11.7 %, 12.6 %, 14.4 %, 30.9 %, 60.7 %) check out. This book’s re-run gives 57.9 % with gender effect-coded and 61.2 % with 0/1 coding, the latter within Monte Carlo error of the paper, which does not state how gender was coded.
  • The size of the winner’s curse. With a true effect d=0.20d = 0.20 and power about 20 %, significant results report on average about 0.45 (an exaggeration of about 2.2 times), not about 0.65 (more than 3 times); the chance of the wrong sign is about 0.5 %.
  • The denominators of replication. The “36 %” of the Open Science Collaboration (2015) is 35 of the 97 originally significant studies, not 36 of 100. Camerer et al.’s 2016 economics replication appeared in Science (11 of 18, 61 %); their 2018 replication of 21 social-science experiments from Nature and Science appeared in Nature Human Behaviour (13, 62 %).
  • The “four-tier” decision rule. The rule in Chapter 8 of this series actually has five verdicts (a benefit of at least the smallest effect of interest, a benefit of undetermined size, harm, no difference worth acting on, unclear), plus prior conditions such as “not interpretable”. The program in Chapter 6 also shows that the strongest verdict, “at least the smallest effect of interest”, appears only about 2.5 % of the time when the true effect equals that smallest effect; the “90 % power” is the probability of reaching some verdict of benefit.
  • Which information theory. “The more possible worlds a statement forbids, the more information it carries” belongs to Bar-Hillel and Carnap’s semantic information theory, which uses logical probability. Shannon’s theory deals with the statistics of signals in communication, and Shannon said explicitly that the semantic aspects of communication are irrelevant to the engineering problem. The two look alike (both take logarithms) but are not the same theory.
  • Titles and sources of the classics. “Seek truth from facts” comes from the account of Liu De, Prince Xian of Hejian, within chapter 53 of the Book of Han, “The Thirteen Sons of Emperor Jing”, not from a separate “Biography of Prince Xian of Hejian”; Yan Shigu’s gloss reads “strive to obtain the facts and always seek what is truly right”. “A single piece of evidence does not establish a conclusion” is Liang Qichao’s summary of the Qianlong–Jiaqing scholarly style in his Intellectual Trends in the Qing Period, not any Qing scholar’s own words; Dai Zhen’s letter to Yao Nai contains “relying on a single piece of evidence to believe it holds generally”, the closest Qing original. Hu Shih’s “bold hypotheses, careful verification” is in his essay “The Methods of Qing Scholars”; “say as much as your evidence allows” is a widely circulated summary, and this book could not find that exact wording in Hu’s writings, so it is treated as a paraphrase.

How to read the boxes

As in other 5D Library books, different kinds of statement are marked separately. Chapter 6 explains why this separation is itself a methodological choice.

Evidence

Evidence boxes report published findings, with sources.

Calculation

Calculation boxes give short Python programs; the text just before each says what it computes. Every program runs as printed in a standard Python 3 installation using only the standard library, with fixed random seeds; its actual output is shown beneath it. The “studies”, “literatures”, “participants” and data in the book are synthesised by the programs and are not any real study.

Hypothesis · not established

Hypothesis boxes hold the author’s ideas, marked “not yet established”, with a statement of what result would show them wrong.

Source text

Source boxes quote classical texts, with the author’s plain-language reading.

Note

Note boxes give cautions and explain choices.

About this edition

This is version 1.0.0, the first complete edition: a preface and eight chapters, in Chinese and English. The book contains 14 Python programs and 4 vector figures. The text, boxes and code were drafted with an AI writing tool from the author’s title, framework and notes; the author is responsible for reviewing and revising them and for checking every source. By this book’s own standards, readers should know that AI-drafted text can get bibliographic details wrong, that the author has a commercial interest in the subject (see below), and that the trial in the book is so far only a protocol. Corrections and missing references are welcome through the 5D Library website.

Disclosure

The author uses the works of 5D Library, including this book, as the methodological basis of his community courses, the 5D Ascension Cards (Dimensional Shift Card Decks) and the Yafu AI knowledge base, and therefore has a commercial interest in the subject of this book. The book promotes no course, card deck or service; the Chapter 8 trial is designed to be run and analysed by researchers independent of the author’s business and does not recruit from the author’s community, students or customers; all the code and the study protocol are published openly for anyone to check.

Fan, Y. (2026). Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis: From Popper's Demarcation and p-Hacking to Ioannidis's Theorem and Pre-Registration, Read Beside Classical Chinese Evidential Scholarship [科学方法论、可证伪性与可重复性危机的数理统计] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23782104 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).