5D Library › Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis · 中文

Chapter 8 Practice and a Pre-Registered Trial

Note

The checklist in this chapter is for judging the strength of evidence; it is not medical or investment advice. If you are using a medicine, supplement or therapy, the checklist’s result should not lead you to start, stop or change it on your own; please discuss it with a doctor or pharmacist.

Part 1 A four-step meta-science checklist for not being fooled

Whenever you meet a claim, whether in a business report, a supplement advertisement or an academic paper, ask four questions in turn.

  1. Falsifiability and an audit of advance prediction. Did the claim state beforehand what result would show it wrong? Was there a protocol registered before the data were collected? If the result were unfavourable, would the claimant “explain it away” with a reason found afterwards? (Chapters 1 and 6)
  2. Screening for multiple outcomes and optional stopping. How many outcomes, subgroups and time points did the study measure? Is the one reported the pre-specified primary outcome, or one picked afterwards? Was the sample size fixed in advance, or was it “run until significant”? (Chapter 3)
  3. Discounting for the small-sample winner’s curse. How many people per group? Was there a control group? If the true effect were only of the size common in the field, what would the study’s power be, and roughly how many times would a significant result be exaggerated? (Chapter 4)
  4. Checking independent replication and disclosure of interests. Has a team unconnected with the original authors repeated the result? Do the claimant or funders have an interest in the conclusion, and have they disclosed it honestly? (Chapter 5)

Evidence

Exaggeration often starts in the press release. Sumner and colleagues analysed 462 biomedical and health press releases issued in 2011 by 20 leading UK universities, with the associated news stories: 40 % of the press releases contained exaggerated advice, 33 % presented correlational research as causal, and 36 % extended conclusions from animal research to humans; when the press release exaggerated, the news followed in 58 %, 81 % and 86 % of cases, against 17 %, 18 % and 10 % when it did not (Sumner et al. 2014). So when checking a news story, the problem often lies not with the journalist but with the press release, or even the paper, behind it.

The checklist applied to three fictional claims, an agency case report (“+300 % conversions”), a supplement advertisement (“clinically proven”) and a pre-registered trial published in a journal, and then to the 5D Library series itself. Step 2 uses 1−0.95k1 – 0.95^k as a rough false-positive risk for kk uncorrected looks at the data (exact only for independent tests); step 3 assumes a true effect of d=0.20d = 0.20.

Calculation

# The four-step checklist on three FICTIONAL claims and on the 5D Library series itself.
# Step 2: 1 - 0.95^k, a rough false-positive risk for k uncorrected looks. Step 3: with SE
# = sqrt(2 / n) a significant d must exceed 1.96 SE; power and exaggeration are for a true
# d = 0.20.
import math
from statistics import NormalDist
N = NormalDist()

def exaggeration(D, s):
    hi, lo = 1 - N.cdf(1.96 - D / s), N.cdf(-1.96 - D / s)
    m_hi, m_lo = D + s * N.pdf(1.96 - D / s) / hi, D - s * N.pdf(-1.96 - D / s) / lo
    return hi + lo, (hi * m_hi - lo * m_lo) / (hi + lo) / D

ITEMS = [  # name, advance prediction + refutation stated, looks k, flexible stopping,
           # n per arm
           # (None = no control group, "plan" = no data yet), independent replications,
           # COI declared
    ("A case report", False, 9, True, None, 0, False),     # agency: "+300 % conversions"
    ("B supplement ad", False, 12, False, 12, 0, False),   # "clinically proven"
    ("C registered RCT", True, 1, False, 400, 1, True),    # pre-registered, in a journal
    ("D this series", True, 1, False, "plan", 0, True),    # 5D Library trial protocols
]
rows = [[], [], [], []]
for name, pred, k, stop, n, rep, coi in ITEMS:
    rows[0].append("yes" if pred else "NO")
    risk = f"{'>' if stop else '~'}{1 - 0.95 ** k:.0%}"
    rows[1].append(f"{k}{'+stop' if stop else ''} {risk}")
    if n is None:
        rows[2].append("no control group")
    elif n == "plan":
        rows[2].append("no data yet")
    else:
        se = math.sqrt(2 / n); pw, ex = exaggeration(0.20, se)
        rows[2].append(f"{n} >{1.96 * se:.2f} {pw:.0%} x{ex:.1f}")
    rows[3].append(f"{rep} / {'yes' if coi else 'NO'}")
print((f"{'step':24s}" + "".join(f"{it[0]:19s}" for it in ITEMS)).rstrip())
for lab, r in zip(("1 predicted in advance", "2 looks -> false pos.",
                   "3 n/arm, d_sig, pw, M", "4 replications / COI"), rows):
    print((f"{lab:24s}" + "".join(f"{x:19s}" for x in r)).rstrip())
print("(step 3: n per arm, smallest significant d, power and exaggeration M"
      " if true d = 0.20)")
print("D also: AI-drafted; author has a commercial interest;"
      " no trial result in the books:")
print("  the protocols can be judged, the interventions cannot yet be.")
# step                    A case report      B supplement ad    C registered RCT   D this series
# 1 predicted in advance  NO                 NO                 yes                yes
# 2 looks -> false pos.   9+stop >37%        12 ~46%            1 ~5%              1 ~5%
# 3 n/arm, d_sig, pw, M   no control group   12 >0.80 8% x4.9   400 >0.14 81% x1.1 no data yet
# 4 replications / COI    0 / NO             0 / NO             1 / yes            0 / yes
# (step 3: n per arm, smallest significant d, power and exaggeration M if true d = 0.20)
# D also: AI-drafted; author has a commercial interest; no trial result in the books:
#   the protocols can be judged, the interventions cannot yet be.

The three fictional claims come out very differently. The case report made no advance prediction, looked at 9 indicators and ended when it chose, giving a false-positive risk above 37 %; more fundamentally, with no control group its effect size cannot be estimated at all. The study behind the supplement advertisement had only 12 per group and picked one significant outcome out of 12, a false-positive risk of about 46 %; at that sample size only estimates above d=0.80d = 0.80 can be significant, and if the true effect were only 0.20 the power would be 8 % and significant results would exaggerate it about 4.9 times. The pre-registered trial in a journal had 400 per group and a single primary outcome, power of 81 % at a true effect of 0.20, an exaggeration ratio of only 1.1, and one independent replication.

A live specimen: the series checks itself

By this checklist, the series passes the first two steps only on paper: as far as this book knows, each book’s Chapter 8 states in advance a single primary outcome and a decision rule, and the hypothesis boxes state what result would show them wrong. But the protocols so far exist only in the books and, as far as this book knows, have not been registered on any trial registry; and a well-written protocol is not evidence. On the last two steps, the series must honestly admit these limitations:

  • No data. The trials in the books are so far only protocols; as far as this book knows, no book reports the result of any trial. What can be judged in these books is the protocols, not the effects of the interventions.
  • Effect-size assumptions lack a prior basis. By the author’s own assessment, most books’ sample-size calculations rest on a chosen smallest effect of interest rather than on published estimates for similar interventions; Chapter 8 of volume 30, for example, states that no published training effect size was available to borrow. The same is true of this book’s trial (see below).
  • What “90 % power” means. The program in Chapter 6 shows that, when the true effect exactly equals the smallest effect of interest, the decision rule reaches its strongest verdict, “at least the smallest effect of interest”, only 2.5 % of the time; the 90 % is the probability of reaching some verdict of benefit.
  • AI drafting. Text, boxes and code were drafted with AI and reviewed by the author, so details may be wrong; checks by the author and AI tools are not independent peer review, still less independent replication.
  • Interests. The author has a commercial interest in the series’ subjects; disclosure lets readers know but does not remove the influence of the interest.
  • Zero independent replications. No independent team has repeated any claim of the series.

These limitations do not mean that the series’ contents are wrong. They mean that, by this book’s own standards, readers should give the series’ hypotheses and trial protocols the same scepticism as any other untested claim.

Part 2 A pre-registered randomised controlled trial

Question

Does a 20-minute training in recognising the garden of forking paths and the small-sample winner’s curse, compared with spending the same time reading a set of conventional critical-thinking maxims, improve adults’ ability to tell news reports that exaggerate because of a methodological flaw from reports that are accurate?

The framework’s primary outcome is “the discrimination d′d' in identifying exaggerated claims when assessing 12 news reports containing methodological flaws”. One point needs adding: discrimination d′d' comes from signal detection theory (volume 26, Chapter 4) and can be computed only with both “signal” and “noise” material; 12 flawed reports alone give only the share of flaws detected (the hit rate), which cannot separate “better at discriminating” from “more suspicious of everything”. This protocol therefore adds to the 12 flawed reports 12 accurate reports matched in topic, length and style.

Evidence

There is a large-trial precedent for teaching people to assess treatment claims. In a cluster-randomised controlled trial in Uganda, 120 primary schools were randomly allocated to the Informed Health Choices curriculum or to control. At the end of the course, children in intervention schools scored on average 62.4 % on a test of assessing the reliability of treatment claims, against 43.1 % in control schools, an adjusted mean difference of 20.0 percentage points; 69 % against 27 % reached the passing mark (at least 13 of 24 correct) (Nsangi et al. 2017). That study involved children, a course far longer than 20 minutes, and a knowledge test rather than discrimination; it shows that such skills can be taught, but cannot be used to estimate this trial’s effect size.

Design

Element Specification
Participants Adults aged 18 to 70, recruited through an independent online survey panel; not recruited from the author’s community, students, customers or card users; excluded are people with professional training in research methods or statistics (more than one introductory course), people working in journalism or research, and anyone who took part in the pilot
Test material 24 short news reports: 12 contain an exaggerated claim caused by a methodological flaw (cherry-picked small-sample subgroups, before–after comparisons with no control group, reporting only the significant one of several outcomes, correlation as causation, animal results extended to humans), 12 report accurately; classified independently before the trial by three methodologists unconnected with the study, keeping only reports on which all three agree; reports that the pilot shows can be told apart by surface cues (headline wording, length) are removed; order of presentation random
Randomisation Individual 1:1 allocation, stratified by education (university degree or not); the allocation sequence is computer-generated by someone not involved in recruitment and revealed only after the participant completes the consent form
Arm A A 20-minute self-paced module: the garden of forking paths (multiple outcomes, subgroups, optional stopping) and the small-sample winner’s curse (significant small-sample effects are too large), taught through four worked examples, with practice questions and feedback; practice material does not overlap the test material
Arm B (active control) 20 minutes reading a set of conventional critical-thinking and media-literacy maxims (check the source and author, look at the date, seek other coverage, beware emotive language), with the same format and number of practice questions as arm A but no teaching of research design or statistics
Blinding Participants are told the study compares “two ways of reading the news” and are not told the hypothesis; judgements are scored automatically by the program; analysts do not know the allocation until the database and analysis code are locked
Primary outcome Discrimination d′d' in “exaggerated / accurate” judgements of the 24 reports immediately after training, with the log-linear correction (add 0.5 to each count and 1 to each total) (Hautus 1995; Macmillan and Creelman 2005)
Secondary outcomes Criterion cc; hit and false-alarm rates; calibration of confidence ratings; d′d' on a parallel set of 24 reports after 4 weeks; a six-item knowledge quiz (also the manipulation check)
Analysis Linear model with arm as the predictor and the stratification variable as covariate; report the difference A − B with its 95 % confidence interval; two-sided α=0.05\alpha = 0.05

There is a concrete reason for making d′d' the primary outcome rather than “how many flawed reports were identified”, which the next program shows.

Sample size

Calculation

# Sample size for the Chapter 8 trial. Primary outcome: discrimination d' when judging 24
# news reports (12 contain an exaggerated claim from a methodological flaw, 12 are
# accurate), with the log-linear correction (add 0.5 to each count, 1 to each total;
# Hautus 1995). ASSUMED, not taken from any study: true d' in the control arm ~ N(1.0,
# 0.5) between people, criterion c ~ N(0, 0.5).
import math, random
from statistics import NormalDist, mean, stdev
N = NormalDist()

def observed_dprime(rng, d, c, n=12):
    h = sum(rng.random() < N.cdf(d / 2 - c) for _ in range(n))
    f = sum(rng.random() < N.cdf(-d / 2 - c) for _ in range(n))
    return N.inv_cdf((h + 0.5) / (n + 1)) - N.inv_cdf((f + 0.5) / (n + 1)), h / n, f / n

def arm(rng, shift_d=0.0, shift_c=0.0, people=20000):
    out = [observed_dprime(rng, rng.gauss(1.0 + shift_d, 0.5), rng.gauss(shift_c, 0.5))
           for _ in range(people)]
    return [o[0] for o in out], mean(o[1] for o in out), mean(o[2] for o in out)

def per_arm(sd, delta, alpha=0.05, power=0.90):
    z = N.inv_cdf(1 - alpha / 2) + N.inv_cdf(power)
    return math.ceil(2 * z ** 2 * sd ** 2 / delta ** 2) + 1

if __name__ == "__main__":
    rng = random.Random(1995)
    b, hb, fb = arm(rng)
    a, ha, fa = arm(rng, shift_d=0.25)
    s, hs, fs = arm(rng, shift_c=-0.4)
    sd = stdev(b)
    print(f"control arm: observed d' mean {mean(b):.2f} (true 1.00),"
          f" SD between people {sd:.2f}")
    print(f"true d' raised by 0.25 -> observed difference {mean(a) - mean(b):.2f}"
          f" (hits {hb:.2f} -> {ha:.2f}, false alarms {fb:.2f} -> {fa:.2f})")
    print(f"only more sceptical (c lowered by 0.4) ->"
          f" d' difference {mean(s) - mean(b):+.2f}"
          f" (hits {hs:.2f}, false alarms {fs:.2f})")
    print("people per arm, 90 % power, alpha 0.05 | observed-d' SESOI 0.20 | 0.25 | 0.30")
    for v in (0.60, round(sd, 2), 0.90):
        ns = " |".join(f"{per_arm(v, dl):5d}" for dl in (0.20, 0.25, 0.30))
        print(f"   SD of observed d' = {v:.2f} {'':16s}|" + ns)
    n = per_arm(round(sd, 2), 0.25)
    rec = math.ceil(n / 0.85)
    print(f"chosen: SESOI 0.25 (observed d'), SD {sd:.2f} -> {n} analysable per arm")
    print(f"allowing 15 % exclusions (attention checks, incomplete sessions)"
          f" -> recruit {rec} per arm,"
          f" {2 * rec} in all")
# control arm: observed d' mean 0.96 (true 1.00), SD between people 0.72
# true d' raised by 0.25 -> observed difference 0.23 (hits 0.67 -> 0.71, false alarms 0.33 -> 0.29)
# only more sceptical (c lowered by 0.4) -> d' difference -0.02 (hits 0.78, false alarms 0.46)
# people per arm, 90 % power, alpha 0.05 | observed-d' SESOI 0.20 | 0.25 | 0.30
#    SD of observed d' = 0.60                 |  191 |  123 |   86
#    SD of observed d' = 0.72                 |  274 |  176 |  123
#    SD of observed d' = 0.90                 |  427 |  274 |  191
# chosen: SESOI 0.25 (observed d'), SD 0.72 -> 176 analysable per arm
# allowing 15 % exclusions (attention checks, incomplete sessions) -> recruit 208 per arm, 416 in all

No published effect size is available to borrow here (the Ugandan trial above differs in population, length of intervention and outcome), so the program states its assumptions explicitly: in the control arm true d′d' is normally distributed across people with mean 1.0 and standard deviation 0.5, and the criterion cc has standard deviation 0.5. Under these assumptions, the observed d′d' from 24 reports has mean 0.96 (the log-linear correction shrinks it slightly towards zero) and a standard deviation between people of 0.72. Raising true d′d' by 0.25 produces an observed difference of 0.23.

The third line of output shows why the primary outcome must be d′d': if a training merely made people more suspicious (lowering the criterion by 0.4), the hit rate would rise from 0.67 to 0.78, looking like “more flaws detected”; but the false-alarm rate would rise at the same time from 0.33 to 0.46, and the difference in d′d' would be only −0.02. Looking only at flawed reports would mistake “suspecting everything” for “discriminating”.

With a smallest effect of interest of 0.25 in observed d′d' (about 0.35 standard deviations) and 90 % power, each arm needs 176 analysable participants; allowing about 15 % exclusions (failed attention checks, incomplete sessions), each arm recruits 208, 416 in all. If the standard deviation were 0.60 or 0.90, the numbers per arm would be 123 and 274; with a smallest effect of 0.20 or 0.30, 274 and 123. By the results of Chapter 6, “90 % power” means that at a true effect of 0.25 the probability of reaching some verdict of benefit is about 90 %; to reach “at least the smallest effect of interest” with good probability, the true effect would need to be about 0.4 to 0.5.

Decision rule

Calculation

# Decision rule fixed before enrolment. D = arm A minus arm B in observed d' (24 reports),
# adjusted for the randomisation stratum (education); 95 % CI; positive = better
# discrimination in A. SESOI 0.25. Gates: more than 25 % of randomised people excluded, or
# the manipulation check fails (A not higher than B on the 6-item knowledge quiz,
# one-sided p < .05) -> NOT INTERPRETABLE.
SESOI = 0.25

def verdict(d, lo, hi, excluded=0.10, manip_ok=True, c_shift=None):
    if excluded > 0.25:
        return f"D = {d:+.2f}: NOT INTERPRETABLE - {excluded:.0%} excluded (limit 25 %)"
    if not manip_ok:
        return f"D = {d:+.2f}: NOT INTERPRETABLE - manipulation check failed"
    if lo >= SESOI:   v = "training improves discrimination by at least 0.25"
    elif lo > 0:      v = "training improves discrimination; size not shown to reach 0.25"
    elif hi < 0:      v = "training WORSENS discrimination"
    elif lo >= -SESOI and hi <= SESOI: v = "no difference worth acting on"
    else:             v = "unclear: the interval is too wide to decide"
    out = f"D = {d:+.2f} ({lo:+.2f} to {hi:+.2f}): {v}"
    if c_shift is not None and abs(c_shift) >= 0.2 and not lo > 0:
        out += (f"\n      note: criterion moved {c_shift:+.2f}"
                " - more suspicion, not more discernment")
    return out

for case in ((0.42, 0.27, 0.57), (0.18, 0.03, 0.33), (0.03, -0.12, 0.18),
             (-0.20, -0.35, -0.05), (0.10, -0.19, 0.39),
             (0.02, -0.13, 0.17, 0.10, True, -0.38),
             (0.40, 0.25, 0.55, 0.31), (0.35, 0.20, 0.50, 0.12, False)):
    print(verdict(*case))
print("secondary outcomes (criterion c, hit and false-alarm rates, confidence ratings,"
      " d' after")
print("  4 weeks) are reported with intervals and never decide the verdict")
# D = +0.42 (+0.27 to +0.57): training improves discrimination by at least 0.25
# D = +0.18 (+0.03 to +0.33): training improves discrimination; size not shown to reach 0.25
# D = +0.03 (-0.12 to +0.18): no difference worth acting on
# D = -0.20 (-0.35 to -0.05): training WORSENS discrimination
# D = +0.10 (-0.19 to +0.39): unclear: the interval is too wide to decide
# D = +0.02 (-0.13 to +0.17): no difference worth acting on
#       note: criterion moved -0.38 - more suspicion, not more discernment
# D = +0.40: NOT INTERPRETABLE - 31% excluded (limit 25 %)
# D = +0.35: NOT INTERPRETABLE - manipulation check failed
# secondary outcomes (criterion c, hit and false-alarm rates, confidence ratings, d' after
#   4 weeks) are reported with intervals and never decide the verdict

The example results are invented. If more than 25 % of those randomised are excluded, or the manipulation check fails (arm A does not score higher than arm B on the knowledge quiz), the report says “not interpretable” whatever the primary outcome. Only if the lower limit of the confidence interval of the difference is at least 0.25 does the report say “the training improved discrimination by at least the smallest effect of interest”. If d′d' shows no difference but the criterion has clearly moved, the report notes specifically that the training made people more suspicious but not more discerning.

What this trial cannot show

First, immediate judgements of selected reports in an experiment cannot show what happens in everyday news reading, still less whether real decisions (such as buying a supplement) change. Second, the 24 reports cover only a few kinds of flaw. Third, even a positive result cannot show which component of the training worked. Fourth, online volunteers may differ from the general population. Fifth, the 4-week test can only partly show whether the effect lasts. Sixth, the result cannot be used to support or oppose any other claim in any book of this series.

Ethics and conflicts of interest

The trial carries little risk, but it must still be approved by an independent ethics committee, participants must give informed consent, and the protocol, the classification of the test material and the analysis code must be registered before the first participant is enrolled (Nosek et al. 2018); where possible it should be submitted as a Registered Report (Chambers 2013), so that publication does not depend on the result. After the trial, all participants are told which reports contained flaws and what the flaws were, and arm B is offered arm A’s training material. Data are stored anonymously.

Disclosure

The author uses the works of 5D Library, including this book, as the methodological basis of his community courses, the 5D Ascension Cards (Dimensional Shift Card Decks) and the Yafu AI knowledge base, and therefore has a commercial interest in the subject of this book. The book promotes no course, card deck or service; the Chapter 8 trial is designed to be run and analysed by researchers independent of the author’s business and does not recruit from the author’s community, students or customers; all the code and the study protocol are published openly for anyone to check. To limit the influence of this interest, the training material, test material and analysis code will be published before enrolment, and the results, data and code will be published whatever the outcome.

Chapter summary

The four-step checklist asks in turn whether a claim was falsifiable in advance, whether it went through multiple comparisons and optional stopping, whether the sample was so small that the effect must be exaggerated, and whether it has been independently replicated with interests disclosed. Applied to three fictional claims, it clearly separates the case report, the supplement advertisement and the pre-registered trial. Applied to the series itself, it shows that the series’ protocols meet the requirements of pre-registration only on paper (they have not been registered), and that there are no data, the effect-size assumptions lack a prior basis, the text is AI-drafted, the author has a commercial interest, and there has been no independent replication. The pre-registered trial has d′d' as its single primary outcome, 176 analysable and 208 recruited per arm, a decision rule written as code, and will be published whatever the result.

References

Chambers, Christopher D. 2013. “Registered Reports: A New Publishing Initiative at Cortex.” Cortex 49 (3): 609–10. https://doi.org/10.1016/j.cortex.2012.12.016.
Hautus, Michael J. 1995. “Corrections for Extreme Proportions and Their Biasing Effects on Estimated Values of d’.” Behavior Research Methods, Instruments, & Computers 27 (1): 46–51.
Macmillan, Neil A., and C. Douglas Creelman. 2005. Detection Theory: A User’s Guide. 2nd ed. Lawrence Erlbaum.
Nosek, Brian A., Charles R. Ebersole, Alexander C. DeHaven, and David T. Mellor. 2018. “The Preregistration Revolution.” Proceedings of the National Academy of Sciences 115 (11): 2600–2606. https://doi.org/10.1073/pnas.1708274114.
Nsangi, Allen, Daniel Semakula, Andrew D. Oxman, et al. 2017. “Effects of the Informed Health Choices Primary School Intervention on the Ability of Children in Uganda to Assess the Reliability of Claims about Treatment Effects: A Cluster-Randomised Controlled Trial.” The Lancet 390 (10092): 374–88. https://doi.org/10.1016/S0140-6736(17)31226-6.
Sumner, Petroc, Solveiga Vivian-Griffiths, Jacky Boivin, et al. 2014. “The Association Between Exaggeration in Health Related Science News and Academic Press Releases: Retrospective Observational Study.” BMJ 349: g7015. https://doi.org/10.1136/bmj.g7015.

Fan, Y. (2026). Scientific Methodology, Falsifiability and the Mathematical Statistics of the Replication Crisis: From Popper's Demarcation and p-Hacking to Ioannidis's Theorem and Pre-Registration, Read Beside Classical Chinese Evidential Scholarship [科学方法论、可证伪性与可重复性危机的数理统计] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23782104 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).