What does A Zero-Leak Socratic Examiner conclude, in numbers, about “Telling It Not to Tell: Answers in Context and the Cumulative Leak Probability”?

Answer: Chapters 1 and 2 quantify why instructions fail: in an illustrative course of 40 items with 3 attempts per item, a class of only 30 leaks at least one item with probability 0.9727 when p is 0.001. They then draw information-flow graphs of four examiner architectures and compute the bit bound of the grading oracle and the cost of attempt caps.

Scope of the numbers: Parameters are illustrative; all calculations are generated by the book's programs, which call no model interface, contain no keys or server addresses and include no concrete extraction prompts. This is not individual business advice.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.9727, 30, 40

Q40 items, k = 3 attempts per student per item
p N P(item) E[leaked] P(any)
0.001030 0.0861 3.44 0.9727
0.0010200 0.4514 18.05 1.0000
0.00101000 0.9503 38.01 1.0000
0.000130 0.0090 0.36 0.3023
0.0001200 0.0582 2.33 0.9093
0.00011000 0.2592 10.37 1.0000
p0.00100: an item is at even odds after 693 attempts on it
p0.00010: an item is at even odds after 6932 attempts on it
p0.00001: an item is at even odds after 69315 attempts on it
tenfold smaller p, tenfold more attempts; the limit stays 1
Show the full program
# Chapter 1. p13: one cohort, many items: the leak is a cohort event.
# A course has Q items whose answers sit in the examiner's context.
# Each of N students makes k probing attempts per item (illustrative).
# A leaked answer can be shared, so the owner's question is not "will
# this student extract it" but "will anyone extract any of them".
#   P(item leaks)  = 1 - (1 - p)^(N k)
#   expected leaked items = Q * P(item leaks)
#   P(any item leaks) = 1 - (1 - p)^(N k Q)
# An instruction that cuts p tenfold only multiplies the attempts
# needed by about ten: it rescales the clock, not the limit.
import math

Q, K = 40, 3


def leak(p, n):
    return 1.0 - math.exp(n * math.log1p(-p))


print("Q = %d items, k = %d attempts per student per item" % (Q, K))
print("p        N      P(item)   E[leaked]  P(any)")
for p in (0.001, 0.0001):
    for n in (30, 200, 1000):
        att = n * K
        per_item = leak(p, att)
        print("%-8.4f %-6d %-9.4f %-10.2f %.4f"
              % (p, n, per_item, Q * per_item, leak(p, att * Q)))
for p in (0.001, 0.0001, 0.00001):
    half = math.log(0.5) / math.log1p(-p)
    print("p = %.5f: an item is at even odds after %d attempts on it"
          % (p, math.ceil(half)))
print("tenfold smaller p, tenfold more attempts; the limit stays 1")

Single source and citation

Fan, Y. (2026). A Zero-Leak Socratic Examiner: Server-Side Isolation of Answers and Bayesian Mastery Probing Trees (Version 1.0.0), Chapter 1: Telling It Not to Tell: Answers in Context and the Cumulative Leak Probability. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-262 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026262,
  author    = {Fan, Yafu},
  title     = {A Zero-Leak Socratic Examiner: Server-Side Isolation of Answers and Bayesian Mastery Probing Trees},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book