What does A Zero-Leak Socratic Examiner conclude, in numbers, about “How Leaks Forge Mastery: Guess Rates, False Certificates and the Blind Spot of Fitting”?
Answer: Program p41 shows that a leak lets learners who hold the answers pass the examiner with certainty, and that a leak reaching a fifth of the class is enough to make one certificate in four false. Program p42 shows that a fit after the fact cannot see the leak: it explains it as a stronger intake with fewer slips, and the curve still fits. Detection therefore relies on structures designed when items are written, namely parallel variants, delayed transfer items and fixed guess rates, and all of them are repairs that come after prevention. The examiner's proper job is to judge mastery quickly and accurately without leaking anything; so, given a string of right-and-wrong bits, which question is the most valuable to ask next? The next chapter builds the probing tree.
Scope of the numbers: Parameters are illustrative; all calculations are generated by the book's programs, which call no model interface, contain no keys or server addresses and include no concrete extraction prompts. This is not individual business advice.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python only
| P(certified): knower 0.9258, guesser 0.0118, key holder 1.0000 | |||
| f | certified | false share | false per 1000 students |
| 0.00 | 0.3774 | 0.0188 | 7.1 |
| 0.05 | 0.4071 | 0.0903 | 36.7 |
| 0.10 | 0.4367 | 0.1520 | 66.4 |
| 0.20 | 0.4960 | 0.2534 | 125.7 |
| 0.40 | 0.6146 | 0.3974 | 244.3 |
Show the full program
# Chapter 4. p41: a leaked key turns into false mastery certificates.
# Examination mode: BKT with L0 = 0.30, T = 0, G = 0.20, S = 0.10 and
# a mastery line of 0.95, at most 6 items per session. Three kinds of
# student (illustrative cohort): knowers (40 per cent, correct with
# probability 0.90), non-knowers without the key (correct 0.20) and
# non-knowers holding a leaked key (correct 1.00). A share f of the
# cohort holds the key. We enumerate all 2^6 answer strings exactly
# and report the false share among those certified as mastered.
import itertools
import math
L0, G, S, LINE, N = 0.30, 0.20, 0.10, 0.95, 6
KNOW = 0.40
def certified(seq):
p = L0
for r in seq:
if r:
p = p * (1 - S) / (p * (1 - S) + (1 - p) * G)
else:
p = p * S / (p * S + (1 - p) * (1 - G))
if p >= LINE:
return True
return False
def p_cert(q):
terms = []
for seq in itertools.product((0, 1), repeat=N):
if certified(seq):
k = len([r for r in seq if r])
terms.append(q ** k * (1 - q) ** (N - k))
return math.fsum(terms)
c_know, c_guess, c_key = p_cert(1 - S), p_cert(G), p_cert(1.0)
print("P(certified): knower %.4f, guesser %.4f, key holder %.4f"
% (c_know, c_guess, c_key))
print("f certified false share false per 1000 students")
for f in (0.0, 0.05, 0.10, 0.20, 0.40):
true_c = KNOW * c_know
# key holders who already know are counted as knowers
false_c = (1 - KNOW) * ((1 - f) * c_guess + f * c_key)
total = true_c + false_c
print("%-7.2f %-10.4f %-12.4f %.1f"
% (f, total, false_c / total, 1000 * false_c))Single source and citation
Fan, Y. (2026). A Zero-Leak Socratic Examiner: Server-Side Isolation of Answers and Bayesian Mastery Probing Trees (Version 1.0.0), Chapter 4: How Leaks Forge Mastery: Guess Rates, False Certificates and the Blind Spot of Fitting. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-262 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026262,
author = {Fan, Yafu},
title = {A Zero-Leak Socratic Examiner: Server-Side Isolation of Answers and Bayesian Mastery Probing Trees},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Telling It Not to Tell: Answers in Context and the Cumulative Leak Probability
- Chapter 2 · Physical Isolation: Server-Side Grading, Information-Flow Graphs and the One-Bit Oracle
- Chapter 3 · Bayesian Knowledge Tracing: Four Parameters, the Mastery Line and Identifiability
- Chapter 5 · The Probing Tree: Information-Gain Item Selection, Greedy Trees and Stopping Rules
- Chapter 6 · Defence in Depth against Prompt Extraction: Correlated Layer Failure, Digest-Only Output Checks and Item Exposure
- Chapter 7 · Classical Sources: "No Opening without Striving" in the Analects and the Slave-Boy Geometry of Plato's Meno
- Chapter 8 · The Empirical Loop: Red-Teaming the Academy Examiner for Leaks, the Mastery Probing Tree and a Pre-Registered Test