5D Library › Diagnostic-to-Pedagogy Conversion Funnels, Mastery-Gated Curriculum Architecture and Certification Network Economics · 中文

Chapter 3. A Five-Question AI Examiner: What Can It Tell Apart?

An exam is a measuring instrument

The author’s design: at the end of each subject an AI examiner sets 5 questions of 4 points each, 20 points in all, with 16 to pass. That asks a learner to earn at least 80% of 20 scoring points. An intuitive view is that a learner with mastery 0.80 should just pass and one below 0.80 should fail. Measurement theory says it is not so simple: any exam is a noisy sample of mastery, and the fewer the questions, the larger the noise (Lord and Novick 1968).

The key assumption of this chapter is correlation within a question: the 4 scoring points of one question are not independent. A question that hits a learner’s blind spot tends to lose all four points; one that hits familiar ground tends to earn all four. The program draws each question’s own success rate from a Beta distribution with mean mm (concentration k=6k = 6), so a question’s score is beta-binomial (Skellam 1948); the total of 5 questions is their exact convolution. For comparison it also computes the binomial distribution for 20 fully independent points.

The pass curve, false passes and false fails

Calculation

# Chapter 3. [p05] The AI examiner: 5 questions, 4 points each, pass 16.
# A candidate's mastery m is the chance of earning any one scoring point.
# Points within a question are correlated: the question's own success
# rate is Beta(k m, k (1 - m)) with k = 6, so a question's score is
# beta-binomial(4). Five questions are independent; the total (0..20)
# is their exact convolution. The pass curve is compared with the naive
# independent-point model Binomial(20, m). Then the candidate population
# is Beta(6, 3) (mean 0.667); "competent" means m >= 0.80. The program
# integrates (grid of 2,000 points) the false-pass share among passers
# and the false-fail share among the competent.
import math

K, PASS = 6.0, 16

def bb_pmf(n, a, b):
    lb = lambda x, y: math.lgamma(x) + math.lgamma(y) - math.lgamma(x + y)
    return [math.exp(math.lgamma(n + 1) - math.lgamma(j + 1)
                     - math.lgamma(n - j + 1)
                     + lb(j + a, n - j + b) - lb(a, b)) for j in range(n + 1)]

def conv(p, q):
    out = [0.0] * (len(p) + len(q) - 1)
    for i, x in enumerate(p):
        for j, y in enumerate(q):
            out[i + j] += x * y
    return out

def total_pmf(m, k=K, nq=5):
    one = bb_pmf(4, k * m, k * (1 - m))
    tot = [1.0]
    for _ in range(nq):
        tot = conv(tot, one)
    return tot

def p_pass(m, k=K, cut=PASS, nq=5):
    return sum(total_pmf(m, k, nq)[cut:])

def p_binom(m, cut=PASS, n=20):
    return sum(math.comb(n, j) * m ** j * (1 - m) ** (n - j)
               for j in range(cut, n + 1))

def beta_pdf(x, a=6.0, b=3.0):
    return x ** (a - 1) * (1 - x) ** (b - 1) * math.gamma(a + b) / (
        math.gamma(a) * math.gamma(b))

if __name__ == "__main__":
    print("   mastery m   P(pass) beta-binomial   P(pass) binomial")
    for m in (0.50, 0.60, 0.70, 0.75, 0.80, 0.85, 0.90, 0.95):
        print("   {:9.2f} {:23.4f} {:18.4f}".format(m, p_pass(m), p_binom(m)))
    n = 2000
    tp = fp = fn = comp = pas = 0.0
    for i in range(n):
        m = (i + 0.5) / n
        w = beta_pdf(m) / n
        pp = p_pass(m)
        pas += w * pp
        if m >= 0.80:
            comp += w
            fn += w * (1 - pp)
        else:
            fp += w * pp
    print("   population Beta(6,3): pass rate {:.4f}".format(pas))
    print("   competent share (m >= 0.80)    : {:.4f}".format(comp))
    print("   false passes among passers     : {:.4f}".format(fp / pas))
    print("   competent who fail             : {:.4f}".format(fn / comp))
    print("\nReading: correlated points make five questions a coarse")
    print("instrument: the curve is flatter than the binomial one.")
    print("Because most candidates sit just below 0.80, almost half of")
    print("all passers are below that line, while about a fifth of the")
    print("competent fail on the day.")
#    mastery m   P(pass) beta-binomial   P(pass) binomial
#         0.50                  0.0182             0.0059
#         0.60                  0.0882             0.0510
#         0.70                  0.2841             0.2375
#         0.75                  0.4410             0.4148
#         0.80                  0.6215             0.6296
#         0.85                  0.7948             0.8298
#         0.90                  0.9240             0.9568
#         0.95                  0.9877             0.9974
#    population Beta(6,3): pass rate 0.3223
#    competent share (m >= 0.80)    : 0.2031
#    false passes among passers     : 0.4930
#    competent who fail             : 0.1953
#
# Reading: correlated points make five questions a coarse
# instrument: the curve is flatter than the binomial one.
# Because most candidates sit just below 0.80, almost half of
# all passers are below that line, while about a fifth of the
# competent fail on the day.
Pass probability against mastery: correlated points (solid) and independent points (dashed). The dotted line is 0.80.

A learner of mastery 0.80 passes with probability only 0.6215; one of 0.70 passes with chance 0.2841; one of 0.90 still fails about 8% of the time (pass probability 0.9240). Correlation within questions flattens the curve at the high end: at 0.90 the independent-point model gives 0.9568, the correlated model only 0.9240.

With the learner population set to Beta(6, 3) (mean mastery about 0.667) and “competent” defined as mastery of at least 0.80, competent learners are 0.2031 of the population and the pass rate is 0.3223. Of the passers, 0.4930 have mastery below 0.80 (false passes); 0.1953 of the competent fail on the day (false fails). False passes are so many because most learners sit not far below 0.80: there are many between 0.70 and 0.80, each with a chance of two or three in ten of passing, and together they make up nearly half the passers.

This is not a fault of the AI examiner but of five questions: any five-question exam, marked by a person or by an AI, has the same coarseness. Chapter 6 looks at the noise of AI marking itself.

Retakes and a leaking item bank

Allowing retakes changes what the exam measures. The program lets a learner sit up to 5 times; each attempt draws 5 questions without replacement from a bank of NN; between attempts the learner studies and mastery rises by 0.03; on a question met before, she remembers the answer and scores full marks with probability 0.8.

Calculation

# Chapter 3. [p06] Retakes, study gains and a leaking item bank.
# The exam of p05 (5 questions, 4 points each, pass at 16, point
# correlation k = 6) may be retaken up to 5 times. Each attempt draws 5
# questions without replacement from a bank of N. Between attempts the
# candidate studies: m rises by g (0.03, capped at 0.97). A question the
# candidate has met before is answered in full if he memorised it (rate
# w = 0.8). Monte Carlo with 20,000 candidates per cell, fixed seed.
# Reported: the share passed by the 1st, 3rd and 5th attempt, and the
# mean number of attempts among those who pass.
import random

K, CUT, TRIES, GAIN, MEMO = 6.0, 16, 5, 0.03, 0.8

def attempt(rng, m, bank, seen, memo):
    score = 0
    for qid in rng.sample(range(bank), 5):
        if qid in seen and rng.random() < memo:
            score += 4
        else:
            q = rng.betavariate(K * m, K * (1 - m))
            score += sum(rng.random() < q for _ in range(4))
        seen.add(qid)
    return score >= CUT

def cohort(rng, m0, bank, gain, memo, n=20000):
    by = [0] * TRIES
    for _ in range(n):
        m, seen = m0, set()
        for t in range(TRIES):
            if attempt(rng, m, bank, seen, memo):
                by[t] += 1
                break
            m = min(0.97, m + gain)
    cum, out = 0, []
    for t in range(TRIES):
        cum += by[t]
        out.append(cum / n)
    mean_t = sum((t + 1) * by[t] for t in range(TRIES)) / max(1, sum(by))
    return out, mean_t

if __name__ == "__main__":
    rng = random.Random(74006)
    print("   m0    bank  gain  memo   by 1st  by 3rd  by 5th  mean tries")
    for m0 in (0.55, 0.65, 0.75):
        for bank, gain, memo in ((400, GAIN, MEMO), (40, GAIN, MEMO),
                                 (40, 0.0, MEMO), (40, 0.0, 0.0)):
            c, mt = cohort(rng, m0, bank, gain, memo)
            print("   {:.2f} {:6d} {:5.2f} {:5.1f} {:8.3f} {:7.3f} {:7.3f}"
                  " {:9.2f}".format(m0, bank, gain, memo, c[0], c[2], c[4],
                                    mt))
    print("\nReading: with a large bank, retakes reward study; with a")
    print("bank of 40, a weak candidate who learns nothing passes")
    print("mostly by remembering questions. The gate measures the bank")
    print("as much as the candidate unless the bank is large or each")
    print("retake draws only unseen questions.")
#    m0    bank  gain  memo   by 1st  by 3rd  by 5th  mean tries
#    0.55    400  0.03   0.8    0.043   0.218   0.517      3.58
#    0.55     40  0.03   0.8    0.041   0.384   0.839      3.54
#    0.55     40  0.00   0.8    0.043   0.277   0.627      3.56
#    0.55     40  0.00   0.0    0.040   0.116   0.188      2.93
#    0.65    400  0.03   0.8    0.164   0.580   0.890      2.90
#    0.65     40  0.03   0.8    0.171   0.718   0.978      2.72
#    0.65     40  0.00   0.8    0.166   0.597   0.889      2.85
#    0.65     40  0.00   0.0    0.168   0.422   0.598      2.63
#    0.75    400  0.03   0.8    0.441   0.916   0.998      1.90
#    0.75     40  0.03   0.8    0.440   0.953   1.000      1.82
#    0.75     40  0.00   0.8    0.446   0.899   0.992      1.92
#    0.75     40  0.00   0.0    0.436   0.824   0.944      1.99
#
# Reading: with a large bank, retakes reward study; with a
# bank of 40, a weak candidate who learns nothing passes
# mostly by remembering questions. The gate measures the bank
# as much as the candidate unless the bank is large or each
# retake draws only unseen questions.

With a bank of 400 and diligent study, a weak candidate starting at 0.55 passes within 5 attempts with share 0.517, which roughly reflects her progress. With a bank of only 40 the share rises to 0.839. The case without study is more telling: starting at 0.55, not studying at all and only remembering questions, she passes within 5 attempts with share 0.627; remembering nothing either, only 0.188. With a bank of 40, then, a weak candidate who learns nothing passes mostly by remembering questions.

A leaking bank hardly affects strong candidates (starting at 0.75, every case passes within 5 attempts with share at least 0.944), so its main effect is to lower the bar for the weak. There are two remedies: a bank large enough for the number of attempts allowed, or retakes that draw only questions the candidate has not seen. Chapter 7 will show that “shown one corner, come back with the other three” is itself a gate made only of unseen questions.

Hypothesis · not established

The author’s interpretation (not yet tested): in the author’s courses, each subject’s bank should hold at least 20 times as many questions as the number of attempts allowed (four times the 5 questions drawn per attempt), retakes should draw only unseen questions, and scripts graded 15 or 16 should be reviewed by a person. What would show this wrong: if, in real exam records, the same candidate scored hardly differently on old and new questions when retaking (so that remembering questions does not help), the requirement on bank size could be relaxed.

Summary

An exam of 5 questions with a pass mark of 16 tells learners apart poorly near mastery 0.80: almost half the passers in the population are below 0.80, and about a fifth of the competent fail on the day. With retakes allowed, a small bank lets weak candidates who learn nothing pass by remembering questions. The exam is a filter, but its mesh is much wider than the nominal “80%”.

References

Lord, Frederic M., and Melvin R. Novick. 1968. Statistical Theories of Mental Test Scores. Addison-Wesley.
Skellam, J. G. 1948. “A Probability Distribution Derived from the Binomial Distribution by Regarding the Probability of Success as Variable between the Sets of Trials.” Journal of the Royal Statistical Society, Series B 10 (2): 257–61.

Fan, Y. (2026). Diagnostic-to-Pedagogy Conversion Funnels, Mastery-Gated Curriculum Architecture and Certification Network Economics: Self-Cannibalization Bounds, Spaced-Retrieval Curves and Two-Sided Practitioner Licensing [诊学一体转化漏斗、掌握度门控课程与持牌人网络经济学] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090197 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).