What does A Zero-Leak Socratic Examiner conclude, in numbers, about “The Empirical Loop: Red-Teaming the Academy Examiner for Leaks, the Mastery Probing Tree and a Pre-Registered Test”?
Answer: Chapter 8 gives the owner's eight-step red-team loop, the distilled Mastery Probing Tree and a pre-registered test; vouching for a term's safety of a build that keeps the answer in context would take 1437950 clean red-team attempts. Marked boxes separate published evidence, runnable Python calculations (21 programs), classical sources and the author's hypotheses.
Scope of the numbers: Parameters are illustrative; all calculations are generated by the book's programs, which call no model interface, contain no keys or server addresses and include no concrete extraction prompts. This is not individual business advice.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 1437950
| red-team n upper p | 3/n cohort risk allowed |
| 100 | 0.0295130 0.0300000 1.0000 |
| 1000 | 0.0029912 0.0030000 1.0000 |
| 10000 | 0.0002995 0.0003000 0.9992 |
| 100000 | 0.0000300 0.0000300 0.5127 |
| 1000000 | 0.0000030 0.0000030 0.0694 |
| to see a leak of rate 0.010 with 0.95 chance: n | 299 |
| to see a leak of rate 0.001 with 0.95 chance: n | 2995 |
| to bound cohort risk at 0.05 the clean run needs n | 1437950 |
| cohort attempts in the p13 course | 24000 |
Show the full program
# Chapter 8. p81: how far a clean red-team run can vouch for a build.
# A red team makes n extraction attempts against a build and records
# zero content leaks. The exact 95 per cent upper bound on the
# per-attempt leak rate is 1 - 0.05^(1/n); the rule of three gives
# about 3/n (Hanley and Lippman-Hand 1983). We then ask what that
# bound still allows over a cohort's real traffic: 200 students x 3
# attempts x 40 items = 24000 attempts (the p13 course).
# For an isolated build a single leaked answer is a counterexample;
# for a build that keeps the answer in context, no finite clean run
# can bring the cohort risk near zero.
import math
COHORT = 200 * 3 * 40
def upper(n):
return 1.0 - 0.05 ** (1.0 / n)
print("red-team n upper p 3/n cohort risk allowed")
for n in (100, 1000, 10000, 100000, 1000000):
u = upper(n)
risk = 1.0 - math.exp(COHORT * math.log1p(-u))
print("%-11d %-9.7f %-9.7f %.4f" % (n, u, 3 / n, risk))
for target in (0.01, 0.001):
n = math.ceil(math.log(0.05) / math.log(1.0 - target))
print("to see a leak of rate %.3f with 0.95 chance: n = %d"
% (target, n))
n = math.ceil(math.log(0.05) / math.log(1.0 - 0.05 / COHORT))
print("to bound cohort risk at 0.05 the clean run needs n = %d" % n)
print("cohort attempts in the p13 course: %d" % COHORT)Single source and citation
Fan, Y. (2026). A Zero-Leak Socratic Examiner: Server-Side Isolation of Answers and Bayesian Mastery Probing Trees (Version 1.0.0), Chapter 8: The Empirical Loop: Red-Teaming the Academy Examiner for Leaks, the Mastery Probing Tree and a Pre-Registered Test. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-262 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026262,
author = {Fan, Yafu},
title = {A Zero-Leak Socratic Examiner: Server-Side Isolation of Answers and Bayesian Mastery Probing Trees},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Telling It Not to Tell: Answers in Context and the Cumulative Leak Probability
- Chapter 2 · Physical Isolation: Server-Side Grading, Information-Flow Graphs and the One-Bit Oracle
- Chapter 3 · Bayesian Knowledge Tracing: Four Parameters, the Mastery Line and Identifiability
- Chapter 4 · How Leaks Forge Mastery: Guess Rates, False Certificates and the Blind Spot of Fitting
- Chapter 5 · The Probing Tree: Information-Gain Item Selection, Greedy Trees and Stopping Rules
- Chapter 6 · Defence in Depth against Prompt Extraction: Correlated Layer Failure, Digest-Only Output Checks and Item Exposure
- Chapter 7 · Classical Sources: "No Opening without Striving" in the Analects and the Slave-Boy Geometry of Plato's Meno