What does A Zero-Leak Socratic Examiner conclude, in numbers, about “How Leaks Forge Mastery: Guess Rates, False Certificates and the Blind Spot of Fitting”?

Answer: Program p41 shows that a leak lets learners who hold the answers pass the examiner with certainty, and that a leak reaching a fifth of the class is enough to make one certificate in four false. Program p42 shows that a fit after the fact cannot see the leak: it explains it as a stronger intake with fewer slips, and the curve still fits. Detection therefore relies on structures designed when items are written, namely parallel variants, delayed transfer items and fixed guess rates, and all of them are repairs that come after prevention. The examiner's proper job is to judge mastery quickly and accurately without leaking anything; so, given a string of right-and-wrong bits, which question is the most valuable to ask next? The next chapter builds the probing tree.

Scope of the numbers: Parameters are illustrative; all calculations are generated by the book's programs, which call no model interface, contain no keys or server addresses and include no concrete extraction prompts. This is not individual business advice.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python only

P(certified): knower 0.9258, guesser 0.0118, key holder 1.0000
fcertifiedfalse sharefalse per 1000 students
0.000.37740.01887.1
0.050.40710.090336.7
0.100.43670.152066.4
0.200.49600.2534125.7
0.400.61460.3974244.3
Show the full program
# Chapter 4. p41: a leaked key turns into false mastery certificates.
# Examination mode: BKT with L0 = 0.30, T = 0, G = 0.20, S = 0.10 and
# a mastery line of 0.95, at most 6 items per session. Three kinds of
# student (illustrative cohort): knowers (40 per cent, correct with
# probability 0.90), non-knowers without the key (correct 0.20) and
# non-knowers holding a leaked key (correct 1.00). A share f of the
# cohort holds the key. We enumerate all 2^6 answer strings exactly
# and report the false share among those certified as mastered.
import itertools
import math

L0, G, S, LINE, N = 0.30, 0.20, 0.10, 0.95, 6
KNOW = 0.40


def certified(seq):
    p = L0
    for r in seq:
        if r:
            p = p * (1 - S) / (p * (1 - S) + (1 - p) * G)
        else:
            p = p * S / (p * S + (1 - p) * (1 - G))
        if p >= LINE:
            return True
    return False


def p_cert(q):
    terms = []
    for seq in itertools.product((0, 1), repeat=N):
        if certified(seq):
            k = len([r for r in seq if r])
            terms.append(q ** k * (1 - q) ** (N - k))
    return math.fsum(terms)


c_know, c_guess, c_key = p_cert(1 - S), p_cert(G), p_cert(1.0)
print("P(certified): knower %.4f, guesser %.4f, key holder %.4f"
      % (c_know, c_guess, c_key))
print("f       certified  false share  false per 1000 students")
for f in (0.0, 0.05, 0.10, 0.20, 0.40):
    true_c = KNOW * c_know
    # key holders who already know are counted as knowers
    false_c = (1 - KNOW) * ((1 - f) * c_guess + f * c_key)
    total = true_c + false_c
    print("%-7.2f %-10.4f %-12.4f %.1f"
          % (f, total, false_c / total, 1000 * false_c))

Single source and citation

Fan, Y. (2026). A Zero-Leak Socratic Examiner: Server-Side Isolation of Answers and Bayesian Mastery Probing Trees (Version 1.0.0), Chapter 4: How Leaks Forge Mastery: Guess Rates, False Certificates and the Blind Spot of Fitting. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-262 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026262,
  author    = {Fan, Yafu},
  title     = {A Zero-Leak Socratic Examiner: Server-Side Isolation of Answers and Bayesian Mastery Probing Trees},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book