What does Machine Prose and Human Cadence conclude, in numbers, about “Why a Detector Score Is Not Evidence: False Positives, Bias and Near-Chance Human Judgment”?

Answer: Detectors are unreliable; published studies say so again and again. Program p31 showed that even with respectable ranking ability, most flagged drafts can be innocent when the base rate is low. Program p32 showed that a single threshold can multiply the false-positive rate more than tenfold for writers with a narrower vocabulary. Program p33 showed that human judgment needs hundreds of trials before it can be told apart from chance.

Scope of the numbers: All examples are synthetic text written by the author or public-domain text, parameters are illustrative, and every calculation is generated by the book's programs. Marked boxes separate published evidence, runnable Python calculations (21 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python only

TPRFPRprevalencePPVwrong flags per 1,000 honest
0.800.100.020.140100.0
0.800.100.100.471100.0
0.800.100.300.774100.0
0.900.050.020.26950.0
0.900.050.100.66750.0
0.900.050.300.88550.0
0.950.010.020.66010.0
0.950.010.100.91310.0
0.950.010.300.97610.0
30 honest texts, FPR 0.05: P(at least one flag) = 0.785
expected honest writers flagged in 30: 1.50
log-odds a flag adds at TPR 0.90, FPR 0.10: 2.197
Show the full program
# Chapter 3. p31: what a detector flag is worth (Bayes' rule).
# A detector with true-positive rate TPR and false-positive rate FPR is
# run on a pool in which a share pi of texts is machine-written.
# PPV = TPR pi / (TPR pi + FPR (1 - pi)) is the chance that a flagged
# text really is machine-written; "wrong per 1,000 honest" counts
# texts by human writers flagged per 1,000 such texts. The three
# operating points are illustrative, not the figures of any product.
import math

POINTS = [(0.80, 0.10), (0.90, 0.05), (0.95, 0.01)]
PREV = [0.02, 0.10, 0.30]
print("TPR   FPR   prevalence    PPV   wrong flags per 1,000 honest")
for tpr, fpr in POINTS:
    for pi in PREV:
        ppv = tpr * pi / (tpr * pi + fpr * (1 - pi))
        print("{:.2f}  {:.2f}     {:.2f}      {:.3f}   {:>6.1f}".format(
            tpr, fpr, pi, ppv, 1000 * fpr))
# accusation arithmetic for one class of 30 honest writers
n, fpr = 30, 0.05
p_any = 1 - (1 - fpr) ** n
print("30 honest texts, FPR 0.05: P(at least one flag) = {:.3f}".format(
    p_any))
print("expected honest writers flagged in 30: {:.2f}".format(n * fpr))
odds = math.log(0.9 / 0.1)
print("log-odds a flag adds at TPR 0.90, FPR 0.10: {:.3f}".format(odds))

Single source and citation

Fan, Y. (2026). Stripping Statistical Fingerprints of Machine Prose, Sentence-Length Burstiness Reconstruction and Master-Level Stylistic Cadence (Version 1.0.0), Chapter 3: Why a Detector Score Is Not Evidence: False Positives, Bias and Near-Chance Human Judgment. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-394 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026394,
  author    = {Fan, Yafu},
  title     = {Stripping Statistical Fingerprints of Machine Prose, Sentence-Length Burstiness Reconstruction and Master-Level Stylistic Cadence},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book