What does A Zero-Leak Socratic Examiner conclude, in numbers, about “The Empirical Loop: Red-Teaming the Academy Examiner for Leaks, the Mastery Probing Tree and a Pre-Registered Test”?

Answer: Chapter 8 gives the owner's eight-step red-team loop, the distilled Mastery Probing Tree and a pre-registered test; vouching for a term's safety of a build that keeps the answer in context would take 1437950 clean red-team attempts. Marked boxes separate published evidence, runnable Python calculations (21 programs), classical sources and the author's hypotheses.

Scope of the numbers: Parameters are illustrative; all calculations are generated by the book's programs, which call no model interface, contain no keys or server addresses and include no concrete extraction prompts. This is not individual business advice.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 1437950

red-team n upper p3/n cohort risk allowed
1000.0295130 0.0300000 1.0000
10000.0029912 0.0030000 1.0000
100000.0002995 0.0003000 0.9992
1000000.0000300 0.0000300 0.5127
10000000.0000030 0.0000030 0.0694
to see a leak of rate 0.010 with 0.95 chance: n299
to see a leak of rate 0.001 with 0.95 chance: n2995
to bound cohort risk at 0.05 the clean run needs n1437950
cohort attempts in the p13 course24000
Show the full program
# Chapter 8. p81: how far a clean red-team run can vouch for a build.
# A red team makes n extraction attempts against a build and records
# zero content leaks. The exact 95 per cent upper bound on the
# per-attempt leak rate is 1 - 0.05^(1/n); the rule of three gives
# about 3/n (Hanley and Lippman-Hand 1983). We then ask what that
# bound still allows over a cohort's real traffic: 200 students x 3
# attempts x 40 items = 24000 attempts (the p13 course).
# For an isolated build a single leaked answer is a counterexample;
# for a build that keeps the answer in context, no finite clean run
# can bring the cohort risk near zero.
import math

COHORT = 200 * 3 * 40


def upper(n):
    return 1.0 - 0.05 ** (1.0 / n)


print("red-team n  upper p   3/n       cohort risk allowed")
for n in (100, 1000, 10000, 100000, 1000000):
    u = upper(n)
    risk = 1.0 - math.exp(COHORT * math.log1p(-u))
    print("%-11d %-9.7f %-9.7f %.4f" % (n, u, 3 / n, risk))
for target in (0.01, 0.001):
    n = math.ceil(math.log(0.05) / math.log(1.0 - target))
    print("to see a leak of rate %.3f with 0.95 chance: n = %d"
          % (target, n))
n = math.ceil(math.log(0.05) / math.log(1.0 - 0.05 / COHORT))
print("to bound cohort risk at 0.05 the clean run needs n = %d" % n)
print("cohort attempts in the p13 course: %d" % COHORT)

Single source and citation

Fan, Y. (2026). A Zero-Leak Socratic Examiner: Server-Side Isolation of Answers and Bayesian Mastery Probing Trees (Version 1.0.0), Chapter 8: The Empirical Loop: Red-Teaming the Academy Examiner for Leaks, the Mastery Probing Tree and a Pre-Registered Test. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-262 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026262,
  author    = {Fan, Yafu},
  title     = {A Zero-Leak Socratic Examiner: Server-Side Isolation of Answers and Bayesian Mastery Probing Trees},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book