5D Library › Autonomic Sympathovagal Balance, Six-Meridian State-Space Models and Multimodal Biomarker Inversion · 中文
Chapter 8. Practice: Accuracy Studies, Test-Retest Reliability and Pre-Registration
How large must an accuracy study be?
To say a screen’s sensitivity is “about 0.80”, one must first answer: within what margin? Only people who really have the condition can be used to estimate sensitivity, so in a study that enrols people consecutively the total needed is that number divided by the prevalence (Buderer 1996).
Calculation
# Chapter 8. p14: How large must a diagnostic accuracy study be? To pin
# sensitivity near 0.80 within +/- d (95%), n_pos = 1.96^2 p (1 - p) / d^2
# people who truly have the condition are needed; likewise n_neg for
# specificity near 0.85. Total enrolment = n_pos / prevalence when the
# study samples consecutive people.
from math import ceil
def n_for(p, d):
return ceil(1.96 ** 2 * p * (1 - p) / d ** 2)
if __name__ == "__main__":
print(" half-width positives negatives enrol at 10% enrol at 20%")
for d in (0.10, 0.05, 0.03):
npos, nneg = n_for(0.80, d), n_for(0.85, d)
e10 = max(ceil(npos / 0.10), ceil(nneg / 0.90))
e20 = max(ceil(npos / 0.20), ceil(nneg / 0.80))
print(" {:10.2f} {:9,d} {:9,d} {:12,d} {:12,d}".format(
d, npos, nneg, e10, e20))
# half-width positives negatives enrol at 10% enrol at 20%
# 0.10 62 49 620 310
# 0.05 246 196 2,460 1,230
# 0.03 683 545 6,830 3,415To pin sensitivity within plus or minus 0.05 needs 246 people who really have the condition; at a prevalence of 10% that means enrolling 2,460 people in all. To narrow it to plus or minus 0.03 the total rises to 6,830. Accuracy studies should report enrolment, reference standard and blinding following the STARD checklist (Bossuyt et al. 2015).
Do three-minute readings repeat?
If a measure differs a lot when the same person is measured again the next day, it means almost nothing for that individual. The intraclass correlation (ICC) measures the share of total variation that comes from differences between people (Shrout and Fleiss 1979).
Calculation
# Chapter 8. p15: Do three-minute readings repeat? Each person has a
# stable true value; each recording adds its own error. Illustrative
# spreads on the log scale (between people / between recordings):
# heart rate 0.12 / 0.05, RMSSD 0.45 / 0.30, LF/HF 0.50 / 0.60. The
# intraclass correlation ICC = between^2 / (between^2 + within^2) is
# estimated from 60 people x 2 recordings (one-way ANOVA), 400 repeats,
# seed 86; averaging k recordings raises it to k ICC / (1 + (k-1) ICC).
import random
MEASURES = (("heart rate", 0.12, 0.05), ("RMSSD", 0.45, 0.30),
("LF/HF", 0.50, 0.60))
def icc_hat(rng, b, w, n=60):
rows = []
for _ in range(n):
t = rng.gauss(0, b)
rows.append((t + rng.gauss(0, w), t + rng.gauss(0, w)))
grand = sum(a + c for a, c in rows) / (2 * n)
msb = 2 * sum(((a + c) / 2 - grand) ** 2 for a, c in rows) / (n - 1)
msw = sum((a - c) ** 2 / 2 for a, c in rows) / n
return (msb - msw) / (msb + msw)
if __name__ == "__main__":
rng = random.Random(86)
print(" measure true ICC estimated 90% range ICC of 3")
for name, b, w in MEASURES:
true = b * b / (b * b + w * w)
est = sorted(icc_hat(rng, b, w) for _ in range(400))
k3 = 3 * true / (1 + 2 * true)
print(" {:<11} {:8.3f} {:9.3f} {:.3f}-{:.3f} {:8.3f}".format(
name, true, sum(est) / 400, est[20], est[379], k3))
# measure true ICC estimated 90% range ICC of 3
# heart rate 0.852 0.850 0.783-0.901 0.945
# RMSSD 0.692 0.692 0.579-0.797 0.871
# LF/HF 0.410 0.405 0.213-0.578 0.676Heart rate has an ICC of about 0.852, fairly reliable; RMSSD has 0.692; LF/HF only 0.410, with a 90% range from 0.213 to 0.578. Averaging three recordings raises the ICC of LF/HF to 0.676, still below a single heart-rate reading. Chapter 1 showed that LF/HF depends strongly on breathing; here is one more point: its test-retest reliability is also the worst.
A pre-registered paired comparison
Finally, design a study to test the hypothesis of Chapter 5: every participant receives both the questionnaire alone and the three-modality screen, with a clinical assessment blind to the screen results as reference. The accuracy of the two methods is compared with McNemar’s test, which depends only on the people on whom the two methods disagree (McNemar 1947).
Calculation
# Chapter 8. p16: A pre-registered paired comparison. Every participant
# gets both the questionnaire alone and the three-modality screen of p09;
# the reference is a blinded clinical assessment. Accuracy is compared
# with McNemar's test, which depends only on the discordant pairs:
# p10 = questionnaire wrong and screen right, p01 = the reverse. Their
# shares are taken from p09's simulated people; then
# n = (1.96 sqrt(pd) + 0.84 sqrt(pd - dl^2))^2 / dl^2, pd = p10 + p01,
# dl = p10 - p01 (two-sided alpha 0.05, power 0.80).
import importlib.util
import random
from math import ceil, sqrt
def load(name):
spec = importlib.util.spec_from_file_location(name, name + ".py")
m = importlib.util.module_from_spec(spec)
spec.loader.exec_module(m)
return m
p09 = load("p09_three_modes")
def top(x, use):
p = p09.post(x, use)
return max(p, key=p.get)
if __name__ == "__main__":
rng = random.Random(86)
a = b = n = 0
for _ in range(10000):
for c in "ABC":
x = p09.draw(rng, c)
q_ok, s_ok = top(x, (0,)) == c, top(x, (0, 1, 2)) == c
a += (not q_ok) and s_ok
b += q_ok and not s_ok
n += 1
p10, p01 = a / n, b / n
pd, dl = p10 + p01, p10 - p01
need = ceil((1.96 * sqrt(pd) + 0.84 * sqrt(pd - dl * dl)) ** 2
/ dl ** 2)
print(" discordant: questionnaire wrong, screen right {:.3f}".format(
p10))
print(" questionnaire right, screen wrong {:.3f}".format(
p01))
print(" participants needed: {}".format(need))
# discordant: questionnaire wrong, screen right 0.319
# questionnaire right, screen wrong 0.084
# participants needed: 55Under the illustrative settings of Chapter 5, the share where the questionnaire is wrong and the screen right is 0.319, and the reverse 0.084; only 55 participants give an eight-in-ten chance of seeing the difference. The real difference is likely much smaller, so the registered protocol should base its sample size on more conservative shares of disagreement.
Note
Key points of pre-registration: before collecting data, publicly register the primary end point (the difference in accuracy between the three-modality screen and the questionnaire alone), the reference standard, the sample-size calculation, the exclusion criteria and the analysis; publish the results whether positive or negative; register the hypotheses of Chapters 4 and 7 separately.
Hypothesis · not established
The author’s interpretation (not yet tested): a pre-registered paired study with a blinded clinical assessment as reference will show the three-modality screen to be more accurate than the questionnaire alone. What would show it wrong: the study’s McNemar test is not significant, or among the discordant pairs the share where the questionnaire is right and the screen wrong is no lower than the reverse.
Summary
A credible accuracy study needs hundreds to thousands of participants; among three-minute readings heart rate is the most stable and LF/HF the least; testing a multimodal screen with a pre-registered paired design needs a sample size set by how often the two methods disagree. Every number in this book is a simulation under illustrative settings, and until these studies are done, the book’s mappings remain hypotheses.
References
Fan, Y. (2026). Autonomic Sympathovagal Balance, Six-Meridian State-Space Models and Multimodal Biomarker Inversion: Heart-Rate Variability Spectral Decomposition, Thermoregulatory Phase Drift and Bayesian Syndrome Classification [自主神经交感迷走平衡、六经辨证状态空间与多模态标志物反演] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090227 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).