What does Funnels, Mastery Gates and Licensing conclude, in numbers, about “A Five-Question AI Examiner: What Can It Tell Apart?”?

Answer: Chapter 3 shows that the five-question examiner tells learners apart poorly near mastery 0.80: 0.4930 of passers in the population are below 0.80 and 0.1953 of the competent fail on the day; with a bank of 40 questions, a weak candidate who learns nothing passes within five attempts with chance 0.627.

Scope of the numbers: Marked boxes separate evidence, 16 runnable Python programs, classical sources and hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.1953, 0.4930, 0.80

mastery m P(pass) beta-binomial P(pass) binomial
0.500.0182 0.0059
0.600.0882 0.0510
0.700.2841 0.2375
0.750.4410 0.4148
0.800.6215 0.6296
0.850.7948 0.8298
0.900.9240 0.9568
0.950.9877 0.9974
population Beta(6,3): pass rate 0.3223
competent share (m >0.80) : 0.2031
false passes among passers0.4930
competent who fail0.1953
Reading: correlated points make five questions a coarse
instrument: the curve is flatter than the binomial one.
Because most candidates sit just below 0.80, almost half of
all passers are below that line, while about a fifth of the
competent fail on the day.
Show the full program
# Chapter 3. [p05] The AI examiner: 5 questions, 4 points each, pass 16.
# A candidate's mastery m is the chance of earning any one scoring point.
# Points within a question are correlated: the question's own success
# rate is Beta(k m, k (1 - m)) with k = 6, so a question's score is
# beta-binomial(4). Five questions are independent; the total (0..20)
# is their exact convolution. The pass curve is compared with the naive
# independent-point model Binomial(20, m). Then the candidate population
# is Beta(6, 3) (mean 0.667); "competent" means m >= 0.80. The program
# integrates (grid of 2,000 points) the false-pass share among passers
# and the false-fail share among the competent.
import math

K, PASS = 6.0, 16

def bb_pmf(n, a, b):
    lb = lambda x, y: math.lgamma(x) + math.lgamma(y) - math.lgamma(x + y)
    return [math.exp(math.lgamma(n + 1) - math.lgamma(j + 1)
                     - math.lgamma(n - j + 1)
                     + lb(j + a, n - j + b) - lb(a, b)) for j in range(n + 1)]

def conv(p, q):
    out = [0.0] * (len(p) + len(q) - 1)
    for i, x in enumerate(p):
        for j, y in enumerate(q):
            out[i + j] += x * y
    return out

def total_pmf(m, k=K, nq=5):
    one = bb_pmf(4, k * m, k * (1 - m))
    tot = [1.0]
    for _ in range(nq):
        tot = conv(tot, one)
    return tot

def p_pass(m, k=K, cut=PASS, nq=5):
    return sum(total_pmf(m, k, nq)[cut:])

def p_binom(m, cut=PASS, n=20):
    return sum(math.comb(n, j) * m ** j * (1 - m) ** (n - j)
               for j in range(cut, n + 1))

def beta_pdf(x, a=6.0, b=3.0):
    return x ** (a - 1) * (1 - x) ** (b - 1) * math.gamma(a + b) / (
        math.gamma(a) * math.gamma(b))

if __name__ == "__main__":
    print("   mastery m   P(pass) beta-binomial   P(pass) binomial")
    for m in (0.50, 0.60, 0.70, 0.75, 0.80, 0.85, 0.90, 0.95):
        print("   {:9.2f} {:23.4f} {:18.4f}".format(m, p_pass(m), p_binom(m)))
    n = 2000
    tp = fp = fn = comp = pas = 0.0
    for i in range(n):
        m = (i + 0.5) / n
        w = beta_pdf(m) / n
        pp = p_pass(m)
        pas += w * pp
        if m >= 0.80:
            comp += w
            fn += w * (1 - pp)
        else:
            fp += w * pp
    print("   population Beta(6,3): pass rate {:.4f}".format(pas))
    print("   competent share (m >= 0.80)    : {:.4f}".format(comp))
    print("   false passes among passers     : {:.4f}".format(fp / pas))
    print("   competent who fail             : {:.4f}".format(fn / comp))
    print("\nReading: correlated points make five questions a coarse")
    print("instrument: the curve is flatter than the binomial one.")
    print("Because most candidates sit just below 0.80, almost half of")
    print("all passers are below that line, while about a fifth of the")
    print("competent fail on the day.")

Single source and citation

Fan, Y. (2026). Diagnostic-to-Pedagogy Conversion Funnels, Mastery-Gated Curriculum Architecture and Certification Network Economics (Version 1.0.0), Chapter 3: A Five-Question AI Examiner: What Can It Tell Apart?. 5D Library. https://doi.org/10.5281/zenodo.24090197

DOI: 10.5281/zenodo.24090197 · Read the monograph · 5DL-B-074 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026074,
  author    = {Fan, Yafu},
  title     = {Diagnostic-to-Pedagogy Conversion Funnels, Mastery-Gated Curriculum Architecture and Certification Network Economics},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.24090197},
  url       = {https://doi.org/10.5281/zenodo.24090197}
}

Other chapter answers in this book