What does Funnels, Mastery Gates and Licensing conclude, in numbers, about “A Five-Question AI Examiner: What Can It Tell Apart?”?
Answer: Chapter 3 shows that the five-question examiner tells learners apart poorly near mastery 0.80: 0.4930 of passers in the population are below 0.80 and 0.1953 of the competent fail on the day; with a bank of 40 questions, a weak candidate who learns nothing passes within five attempts with chance 0.627.
Scope of the numbers: Marked boxes separate evidence, 16 runnable Python programs, classical sources and hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.1953, 0.4930, 0.80
| mastery m P(pass) beta-binomial P(pass) binomial | |
| 0.50 | 0.0182 0.0059 |
| 0.60 | 0.0882 0.0510 |
| 0.70 | 0.2841 0.2375 |
| 0.75 | 0.4410 0.4148 |
| 0.80 | 0.6215 0.6296 |
| 0.85 | 0.7948 0.8298 |
| 0.90 | 0.9240 0.9568 |
| 0.95 | 0.9877 0.9974 |
| population Beta(6,3): pass rate 0.3223 | |
| competent share (m > | 0.80) : 0.2031 |
| false passes among passers | 0.4930 |
| competent who fail | 0.1953 |
| Reading: correlated points make five questions a coarse | |
| instrument: the curve is flatter than the binomial one. | |
| Because most candidates sit just below 0.80, almost half of | |
| all passers are below that line, while about a fifth of the | |
| competent fail on the day. |
Show the full program
# Chapter 3. [p05] The AI examiner: 5 questions, 4 points each, pass 16.
# A candidate's mastery m is the chance of earning any one scoring point.
# Points within a question are correlated: the question's own success
# rate is Beta(k m, k (1 - m)) with k = 6, so a question's score is
# beta-binomial(4). Five questions are independent; the total (0..20)
# is their exact convolution. The pass curve is compared with the naive
# independent-point model Binomial(20, m). Then the candidate population
# is Beta(6, 3) (mean 0.667); "competent" means m >= 0.80. The program
# integrates (grid of 2,000 points) the false-pass share among passers
# and the false-fail share among the competent.
import math
K, PASS = 6.0, 16
def bb_pmf(n, a, b):
lb = lambda x, y: math.lgamma(x) + math.lgamma(y) - math.lgamma(x + y)
return [math.exp(math.lgamma(n + 1) - math.lgamma(j + 1)
- math.lgamma(n - j + 1)
+ lb(j + a, n - j + b) - lb(a, b)) for j in range(n + 1)]
def conv(p, q):
out = [0.0] * (len(p) + len(q) - 1)
for i, x in enumerate(p):
for j, y in enumerate(q):
out[i + j] += x * y
return out
def total_pmf(m, k=K, nq=5):
one = bb_pmf(4, k * m, k * (1 - m))
tot = [1.0]
for _ in range(nq):
tot = conv(tot, one)
return tot
def p_pass(m, k=K, cut=PASS, nq=5):
return sum(total_pmf(m, k, nq)[cut:])
def p_binom(m, cut=PASS, n=20):
return sum(math.comb(n, j) * m ** j * (1 - m) ** (n - j)
for j in range(cut, n + 1))
def beta_pdf(x, a=6.0, b=3.0):
return x ** (a - 1) * (1 - x) ** (b - 1) * math.gamma(a + b) / (
math.gamma(a) * math.gamma(b))
if __name__ == "__main__":
print(" mastery m P(pass) beta-binomial P(pass) binomial")
for m in (0.50, 0.60, 0.70, 0.75, 0.80, 0.85, 0.90, 0.95):
print(" {:9.2f} {:23.4f} {:18.4f}".format(m, p_pass(m), p_binom(m)))
n = 2000
tp = fp = fn = comp = pas = 0.0
for i in range(n):
m = (i + 0.5) / n
w = beta_pdf(m) / n
pp = p_pass(m)
pas += w * pp
if m >= 0.80:
comp += w
fn += w * (1 - pp)
else:
fp += w * pp
print(" population Beta(6,3): pass rate {:.4f}".format(pas))
print(" competent share (m >= 0.80) : {:.4f}".format(comp))
print(" false passes among passers : {:.4f}".format(fp / pas))
print(" competent who fail : {:.4f}".format(fn / comp))
print("\nReading: correlated points make five questions a coarse")
print("instrument: the curve is flatter than the binomial one.")
print("Because most candidates sit just below 0.80, almost half of")
print("all passers are below that line, while about a fifth of the")
print("competent fail on the day.")Single source and citation
Fan, Y. (2026). Diagnostic-to-Pedagogy Conversion Funnels, Mastery-Gated Curriculum Architecture and Certification Network Economics (Version 1.0.0), Chapter 3: A Five-Question AI Examiner: What Can It Tell Apart?. 5D Library. https://doi.org/10.5281/zenodo.24090197
DOI: 10.5281/zenodo.24090197 · Read the monograph · 5DL-B-074 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026074,
author = {Fan, Yafu},
title = {Diagnostic-to-Pedagogy Conversion Funnels, Mastery-Gated Curriculum Architecture and Certification Network Economics},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.24090197},
url = {https://doi.org/10.5281/zenodo.24090197}
}Other chapter answers in this book
- Chapter 1 · The Self-Cannibalization Bound: When Does a Course Pay?
- Chapter 2 · Prerequisite Graphs and Forgetting Curves: The Skeleton of a Course
- Chapter 4 · Chapter Granularity, Dropout and Twelve Gates
- Chapter 5 · The Pass Mark as a Quality Filter: Static Optimum and Reputation
- Chapter 6 · Two-Sided Networks, Grading Noise and Teaching to the Test
- Chapter 7 · "No Opening for One Not Eager", Graded Examinations and Guild Heads
- Chapter 8 · An Audit of Course and Licence, and a Pre-Registered Experiment