5D Library › Diagnostic-to-Pedagogy Conversion Funnels, Mastery-Gated Curriculum Architecture and Certification Network Economics · 中文
Chapter 3. A Five-Question AI Examiner: What Can It Tell Apart?
An exam is a measuring instrument
The author’s design: at the end of each subject an AI examiner sets 5 questions of 4 points each, 20 points in all, with 16 to pass. That asks a learner to earn at least 80% of 20 scoring points. An intuitive view is that a learner with mastery 0.80 should just pass and one below 0.80 should fail. Measurement theory says it is not so simple: any exam is a noisy sample of mastery, and the fewer the questions, the larger the noise (Lord and Novick 1968).
The key assumption of this chapter is correlation within a question: the 4 scoring points of one question are not independent. A question that hits a learner’s blind spot tends to lose all four points; one that hits familiar ground tends to earn all four. The program draws each question’s own success rate from a Beta distribution with mean (concentration ), so a question’s score is beta-binomial (Skellam 1948); the total of 5 questions is their exact convolution. For comparison it also computes the binomial distribution for 20 fully independent points.
The pass curve, false passes and false fails
Calculation
# Chapter 3. [p05] The AI examiner: 5 questions, 4 points each, pass 16.
# A candidate's mastery m is the chance of earning any one scoring point.
# Points within a question are correlated: the question's own success
# rate is Beta(k m, k (1 - m)) with k = 6, so a question's score is
# beta-binomial(4). Five questions are independent; the total (0..20)
# is their exact convolution. The pass curve is compared with the naive
# independent-point model Binomial(20, m). Then the candidate population
# is Beta(6, 3) (mean 0.667); "competent" means m >= 0.80. The program
# integrates (grid of 2,000 points) the false-pass share among passers
# and the false-fail share among the competent.
import math
K, PASS = 6.0, 16
def bb_pmf(n, a, b):
lb = lambda x, y: math.lgamma(x) + math.lgamma(y) - math.lgamma(x + y)
return [math.exp(math.lgamma(n + 1) - math.lgamma(j + 1)
- math.lgamma(n - j + 1)
+ lb(j + a, n - j + b) - lb(a, b)) for j in range(n + 1)]
def conv(p, q):
out = [0.0] * (len(p) + len(q) - 1)
for i, x in enumerate(p):
for j, y in enumerate(q):
out[i + j] += x * y
return out
def total_pmf(m, k=K, nq=5):
one = bb_pmf(4, k * m, k * (1 - m))
tot = [1.0]
for _ in range(nq):
tot = conv(tot, one)
return tot
def p_pass(m, k=K, cut=PASS, nq=5):
return sum(total_pmf(m, k, nq)[cut:])
def p_binom(m, cut=PASS, n=20):
return sum(math.comb(n, j) * m ** j * (1 - m) ** (n - j)
for j in range(cut, n + 1))
def beta_pdf(x, a=6.0, b=3.0):
return x ** (a - 1) * (1 - x) ** (b - 1) * math.gamma(a + b) / (
math.gamma(a) * math.gamma(b))
if __name__ == "__main__":
print(" mastery m P(pass) beta-binomial P(pass) binomial")
for m in (0.50, 0.60, 0.70, 0.75, 0.80, 0.85, 0.90, 0.95):
print(" {:9.2f} {:23.4f} {:18.4f}".format(m, p_pass(m), p_binom(m)))
n = 2000
tp = fp = fn = comp = pas = 0.0
for i in range(n):
m = (i + 0.5) / n
w = beta_pdf(m) / n
pp = p_pass(m)
pas += w * pp
if m >= 0.80:
comp += w
fn += w * (1 - pp)
else:
fp += w * pp
print(" population Beta(6,3): pass rate {:.4f}".format(pas))
print(" competent share (m >= 0.80) : {:.4f}".format(comp))
print(" false passes among passers : {:.4f}".format(fp / pas))
print(" competent who fail : {:.4f}".format(fn / comp))
print("\nReading: correlated points make five questions a coarse")
print("instrument: the curve is flatter than the binomial one.")
print("Because most candidates sit just below 0.80, almost half of")
print("all passers are below that line, while about a fifth of the")
print("competent fail on the day.")
# mastery m P(pass) beta-binomial P(pass) binomial
# 0.50 0.0182 0.0059
# 0.60 0.0882 0.0510
# 0.70 0.2841 0.2375
# 0.75 0.4410 0.4148
# 0.80 0.6215 0.6296
# 0.85 0.7948 0.8298
# 0.90 0.9240 0.9568
# 0.95 0.9877 0.9974
# population Beta(6,3): pass rate 0.3223
# competent share (m >= 0.80) : 0.2031
# false passes among passers : 0.4930
# competent who fail : 0.1953
#
# Reading: correlated points make five questions a coarse
# instrument: the curve is flatter than the binomial one.
# Because most candidates sit just below 0.80, almost half of
# all passers are below that line, while about a fifth of the
# competent fail on the day.A learner of mastery 0.80 passes with probability only 0.6215; one of 0.70 passes with chance 0.2841; one of 0.90 still fails about 8% of the time (pass probability 0.9240). Correlation within questions flattens the curve at the high end: at 0.90 the independent-point model gives 0.9568, the correlated model only 0.9240.
With the learner population set to Beta(6, 3) (mean mastery about 0.667) and “competent” defined as mastery of at least 0.80, competent learners are 0.2031 of the population and the pass rate is 0.3223. Of the passers, 0.4930 have mastery below 0.80 (false passes); 0.1953 of the competent fail on the day (false fails). False passes are so many because most learners sit not far below 0.80: there are many between 0.70 and 0.80, each with a chance of two or three in ten of passing, and together they make up nearly half the passers.
This is not a fault of the AI examiner but of five questions: any five-question exam, marked by a person or by an AI, has the same coarseness. Chapter 6 looks at the noise of AI marking itself.
Retakes and a leaking item bank
Allowing retakes changes what the exam measures. The program lets a learner sit up to 5 times; each attempt draws 5 questions without replacement from a bank of ; between attempts the learner studies and mastery rises by 0.03; on a question met before, she remembers the answer and scores full marks with probability 0.8.
Calculation
# Chapter 3. [p06] Retakes, study gains and a leaking item bank.
# The exam of p05 (5 questions, 4 points each, pass at 16, point
# correlation k = 6) may be retaken up to 5 times. Each attempt draws 5
# questions without replacement from a bank of N. Between attempts the
# candidate studies: m rises by g (0.03, capped at 0.97). A question the
# candidate has met before is answered in full if he memorised it (rate
# w = 0.8). Monte Carlo with 20,000 candidates per cell, fixed seed.
# Reported: the share passed by the 1st, 3rd and 5th attempt, and the
# mean number of attempts among those who pass.
import random
K, CUT, TRIES, GAIN, MEMO = 6.0, 16, 5, 0.03, 0.8
def attempt(rng, m, bank, seen, memo):
score = 0
for qid in rng.sample(range(bank), 5):
if qid in seen and rng.random() < memo:
score += 4
else:
q = rng.betavariate(K * m, K * (1 - m))
score += sum(rng.random() < q for _ in range(4))
seen.add(qid)
return score >= CUT
def cohort(rng, m0, bank, gain, memo, n=20000):
by = [0] * TRIES
for _ in range(n):
m, seen = m0, set()
for t in range(TRIES):
if attempt(rng, m, bank, seen, memo):
by[t] += 1
break
m = min(0.97, m + gain)
cum, out = 0, []
for t in range(TRIES):
cum += by[t]
out.append(cum / n)
mean_t = sum((t + 1) * by[t] for t in range(TRIES)) / max(1, sum(by))
return out, mean_t
if __name__ == "__main__":
rng = random.Random(74006)
print(" m0 bank gain memo by 1st by 3rd by 5th mean tries")
for m0 in (0.55, 0.65, 0.75):
for bank, gain, memo in ((400, GAIN, MEMO), (40, GAIN, MEMO),
(40, 0.0, MEMO), (40, 0.0, 0.0)):
c, mt = cohort(rng, m0, bank, gain, memo)
print(" {:.2f} {:6d} {:5.2f} {:5.1f} {:8.3f} {:7.3f} {:7.3f}"
" {:9.2f}".format(m0, bank, gain, memo, c[0], c[2], c[4],
mt))
print("\nReading: with a large bank, retakes reward study; with a")
print("bank of 40, a weak candidate who learns nothing passes")
print("mostly by remembering questions. The gate measures the bank")
print("as much as the candidate unless the bank is large or each")
print("retake draws only unseen questions.")
# m0 bank gain memo by 1st by 3rd by 5th mean tries
# 0.55 400 0.03 0.8 0.043 0.218 0.517 3.58
# 0.55 40 0.03 0.8 0.041 0.384 0.839 3.54
# 0.55 40 0.00 0.8 0.043 0.277 0.627 3.56
# 0.55 40 0.00 0.0 0.040 0.116 0.188 2.93
# 0.65 400 0.03 0.8 0.164 0.580 0.890 2.90
# 0.65 40 0.03 0.8 0.171 0.718 0.978 2.72
# 0.65 40 0.00 0.8 0.166 0.597 0.889 2.85
# 0.65 40 0.00 0.0 0.168 0.422 0.598 2.63
# 0.75 400 0.03 0.8 0.441 0.916 0.998 1.90
# 0.75 40 0.03 0.8 0.440 0.953 1.000 1.82
# 0.75 40 0.00 0.8 0.446 0.899 0.992 1.92
# 0.75 40 0.00 0.0 0.436 0.824 0.944 1.99
#
# Reading: with a large bank, retakes reward study; with a
# bank of 40, a weak candidate who learns nothing passes
# mostly by remembering questions. The gate measures the bank
# as much as the candidate unless the bank is large or each
# retake draws only unseen questions.With a bank of 400 and diligent study, a weak candidate starting at 0.55 passes within 5 attempts with share 0.517, which roughly reflects her progress. With a bank of only 40 the share rises to 0.839. The case without study is more telling: starting at 0.55, not studying at all and only remembering questions, she passes within 5 attempts with share 0.627; remembering nothing either, only 0.188. With a bank of 40, then, a weak candidate who learns nothing passes mostly by remembering questions.
A leaking bank hardly affects strong candidates (starting at 0.75, every case passes within 5 attempts with share at least 0.944), so its main effect is to lower the bar for the weak. There are two remedies: a bank large enough for the number of attempts allowed, or retakes that draw only questions the candidate has not seen. Chapter 7 will show that “shown one corner, come back with the other three” is itself a gate made only of unseen questions.
Hypothesis · not established
The author’s interpretation (not yet tested): in the author’s courses, each subject’s bank should hold at least 20 times as many questions as the number of attempts allowed (four times the 5 questions drawn per attempt), retakes should draw only unseen questions, and scripts graded 15 or 16 should be reviewed by a person. What would show this wrong: if, in real exam records, the same candidate scored hardly differently on old and new questions when retaking (so that remembering questions does not help), the requirement on bank size could be relaxed.
Summary
An exam of 5 questions with a pass mark of 16 tells learners apart poorly near mastery 0.80: almost half the passers in the population are below 0.80, and about a fifth of the competent fail on the day. With retakes allowed, a small bank lets weak candidates who learn nothing pass by remembering questions. The exam is a filter, but its mesh is much wider than the nominal “80%”.
References
Fan, Y. (2026). Diagnostic-to-Pedagogy Conversion Funnels, Mastery-Gated Curriculum Architecture and Certification Network Economics: Self-Cannibalization Bounds, Spaced-Retrieval Curves and Two-Sided Practitioner Licensing [诊学一体转化漏斗、掌握度门控课程与持牌人网络经济学] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090197 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).