5D Library › 诊学一体转化漏斗、掌握度门控课程与持牌人网络经济学 · English
第三章 五道题的 AI 考官:它能分辨什么
考试是一件测量仪器
作者的设计是:每个科目结束时,AI 考官出 5 道题,每题 4 分,满分 20 分,16 分及格。这相当于要求学员在 20 个评分点上至少拿到 80%。一个直观的看法是:掌握度 0.80 的人应当刚好及格,低于 0.80 的人应当不及格。测量理论告诉我们事情没有这么简单:任何考试都只是对掌握度的一次有噪声的抽样,题量越少,噪声越大 (Lord and Novick 1968)。
本章的关键假设是题内相关:同一道题的 4 个评分点不是独立的。一道题若考到学员的盲区,他往往四个点全丢;考到熟悉之处,往往全得。程序让每道题的成功率服从均值为 的 Beta 分布(集中度 ),于是一道题的得分服从 Beta-二项分布 (Skellam 1948);5 道题的总分是它们的精确卷积。作为对照,也计算 20 个评分点完全独立时的二项分布。
及格曲线、误放与误拒
推导
# Chapter 3. [p05] The AI examiner: 5 questions, 4 points each, pass 16.
# A candidate's mastery m is the chance of earning any one scoring point.
# Points within a question are correlated: the question's own success
# rate is Beta(k m, k (1 - m)) with k = 6, so a question's score is
# beta-binomial(4). Five questions are independent; the total (0..20)
# is their exact convolution. The pass curve is compared with the naive
# independent-point model Binomial(20, m). Then the candidate population
# is Beta(6, 3) (mean 0.667); "competent" means m >= 0.80. The program
# integrates (grid of 2,000 points) the false-pass share among passers
# and the false-fail share among the competent.
import math
K, PASS = 6.0, 16
def bb_pmf(n, a, b):
lb = lambda x, y: math.lgamma(x) + math.lgamma(y) - math.lgamma(x + y)
return [math.exp(math.lgamma(n + 1) - math.lgamma(j + 1)
- math.lgamma(n - j + 1)
+ lb(j + a, n - j + b) - lb(a, b)) for j in range(n + 1)]
def conv(p, q):
out = [0.0] * (len(p) + len(q) - 1)
for i, x in enumerate(p):
for j, y in enumerate(q):
out[i + j] += x * y
return out
def total_pmf(m, k=K, nq=5):
one = bb_pmf(4, k * m, k * (1 - m))
tot = [1.0]
for _ in range(nq):
tot = conv(tot, one)
return tot
def p_pass(m, k=K, cut=PASS, nq=5):
return sum(total_pmf(m, k, nq)[cut:])
def p_binom(m, cut=PASS, n=20):
return sum(math.comb(n, j) * m ** j * (1 - m) ** (n - j)
for j in range(cut, n + 1))
def beta_pdf(x, a=6.0, b=3.0):
return x ** (a - 1) * (1 - x) ** (b - 1) * math.gamma(a + b) / (
math.gamma(a) * math.gamma(b))
if __name__ == "__main__":
print(" mastery m P(pass) beta-binomial P(pass) binomial")
for m in (0.50, 0.60, 0.70, 0.75, 0.80, 0.85, 0.90, 0.95):
print(" {:9.2f} {:23.4f} {:18.4f}".format(m, p_pass(m), p_binom(m)))
n = 2000
tp = fp = fn = comp = pas = 0.0
for i in range(n):
m = (i + 0.5) / n
w = beta_pdf(m) / n
pp = p_pass(m)
pas += w * pp
if m >= 0.80:
comp += w
fn += w * (1 - pp)
else:
fp += w * pp
print(" population Beta(6,3): pass rate {:.4f}".format(pas))
print(" competent share (m >= 0.80) : {:.4f}".format(comp))
print(" false passes among passers : {:.4f}".format(fp / pas))
print(" competent who fail : {:.4f}".format(fn / comp))
print("\nReading: correlated points make five questions a coarse")
print("instrument: the curve is flatter than the binomial one.")
print("Because most candidates sit just below 0.80, almost half of")
print("all passers are below that line, while about a fifth of the")
print("competent fail on the day.")
# mastery m P(pass) beta-binomial P(pass) binomial
# 0.50 0.0182 0.0059
# 0.60 0.0882 0.0510
# 0.70 0.2841 0.2375
# 0.75 0.4410 0.4148
# 0.80 0.6215 0.6296
# 0.85 0.7948 0.8298
# 0.90 0.9240 0.9568
# 0.95 0.9877 0.9974
# population Beta(6,3): pass rate 0.3223
# competent share (m >= 0.80) : 0.2031
# false passes among passers : 0.4930
# competent who fail : 0.1953
#
# Reading: correlated points make five questions a coarse
# instrument: the curve is flatter than the binomial one.
# Because most candidates sit just below 0.80, almost half of
# all passers are below that line, while about a fifth of the
# competent fail on the day.掌握度 0.80 的学员,及格概率只有 0.6215;0.70 的学员有 0.2841 的机会及格,0.90 的学员仍有约 8% 的机会不及格(及格概率 0.9240)。题内相关使曲线在高掌握度一端更平:0.90 时,独立评分点模型给出 0.9568,相关模型只有 0.9240。
把学员人群设为 Beta(6, 3)(平均掌握度约 0.667),并把「胜任」定义为掌握度不低于 0.80,人群中胜任者占 0.2031,及格率 0.3223。其中 0.4930 的及格者掌握度低于 0.80(误放),0.1953 的胜任者当天不及格(误拒)。误放之所以这么高,是因为大多数学员的掌握度正好在 0.80 下方不远:在 0.70—0.80 之间的人很多,每人都有两三成的机会及格,加起来就占了及格者的近一半。
这不是 AI 考官的毛病,而是 5 道题的毛病:任何 5 道题的考试,无论人工还是 AI 阅卷,都有同样的粗糙度。第六章会看 AI 阅卷本身的噪声。
补考与题库泄漏
允许补考会改变考试测的是什么。程序让学员最多考 5 次;每次从 道题的题库中不放回地抽 5 道;两次之间学员复习,掌握度上升 0.03;遇到见过的题,有 0.8 的概率记得答案并拿满分。
推导
# Chapter 3. [p06] Retakes, study gains and a leaking item bank.
# The exam of p05 (5 questions, 4 points each, pass at 16, point
# correlation k = 6) may be retaken up to 5 times. Each attempt draws 5
# questions without replacement from a bank of N. Between attempts the
# candidate studies: m rises by g (0.03, capped at 0.97). A question the
# candidate has met before is answered in full if he memorised it (rate
# w = 0.8). Monte Carlo with 20,000 candidates per cell, fixed seed.
# Reported: the share passed by the 1st, 3rd and 5th attempt, and the
# mean number of attempts among those who pass.
import random
K, CUT, TRIES, GAIN, MEMO = 6.0, 16, 5, 0.03, 0.8
def attempt(rng, m, bank, seen, memo):
score = 0
for qid in rng.sample(range(bank), 5):
if qid in seen and rng.random() < memo:
score += 4
else:
q = rng.betavariate(K * m, K * (1 - m))
score += sum(rng.random() < q for _ in range(4))
seen.add(qid)
return score >= CUT
def cohort(rng, m0, bank, gain, memo, n=20000):
by = [0] * TRIES
for _ in range(n):
m, seen = m0, set()
for t in range(TRIES):
if attempt(rng, m, bank, seen, memo):
by[t] += 1
break
m = min(0.97, m + gain)
cum, out = 0, []
for t in range(TRIES):
cum += by[t]
out.append(cum / n)
mean_t = sum((t + 1) * by[t] for t in range(TRIES)) / max(1, sum(by))
return out, mean_t
if __name__ == "__main__":
rng = random.Random(74006)
print(" m0 bank gain memo by 1st by 3rd by 5th mean tries")
for m0 in (0.55, 0.65, 0.75):
for bank, gain, memo in ((400, GAIN, MEMO), (40, GAIN, MEMO),
(40, 0.0, MEMO), (40, 0.0, 0.0)):
c, mt = cohort(rng, m0, bank, gain, memo)
print(" {:.2f} {:6d} {:5.2f} {:5.1f} {:8.3f} {:7.3f} {:7.3f}"
" {:9.2f}".format(m0, bank, gain, memo, c[0], c[2], c[4],
mt))
print("\nReading: with a large bank, retakes reward study; with a")
print("bank of 40, a weak candidate who learns nothing passes")
print("mostly by remembering questions. The gate measures the bank")
print("as much as the candidate unless the bank is large or each")
print("retake draws only unseen questions.")
# m0 bank gain memo by 1st by 3rd by 5th mean tries
# 0.55 400 0.03 0.8 0.043 0.218 0.517 3.58
# 0.55 40 0.03 0.8 0.041 0.384 0.839 3.54
# 0.55 40 0.00 0.8 0.043 0.277 0.627 3.56
# 0.55 40 0.00 0.0 0.040 0.116 0.188 2.93
# 0.65 400 0.03 0.8 0.164 0.580 0.890 2.90
# 0.65 40 0.03 0.8 0.171 0.718 0.978 2.72
# 0.65 40 0.00 0.8 0.166 0.597 0.889 2.85
# 0.65 40 0.00 0.0 0.168 0.422 0.598 2.63
# 0.75 400 0.03 0.8 0.441 0.916 0.998 1.90
# 0.75 40 0.03 0.8 0.440 0.953 1.000 1.82
# 0.75 40 0.00 0.8 0.446 0.899 0.992 1.92
# 0.75 40 0.00 0.0 0.436 0.824 0.944 1.99
#
# Reading: with a large bank, retakes reward study; with a
# bank of 40, a weak candidate who learns nothing passes
# mostly by remembering questions. The gate measures the bank
# as much as the candidate unless the bank is large or each
# retake draws only unseen questions.题库有 400 题、学员认真复习时,起点 0.55 的弱考生在 5 次之内通过的比例是 0.517,大致反映了他的进步。题库只有 40 题时,这一比例升到 0.839。更说明问题的是不学习的情形:起点 0.55、完全不复习、只记题时,5 次之内通过 0.627;连题也不记时只有 0.188。也就是说,在 40 题的题库下,一位不学习的弱考生大半是靠记题通过的。
题库泄漏对强考生影响很小(起点 0.75 时,各种情形下 5 次之内的通过率都不低于 0.944),因此它主要是放宽了对弱考生的门槛。补救办法有两种:题库足够大(相对于允许的补考次数),或者每次补考只从考生没见过的题中抽题。第七章会看到,「举一隅不以三隅反」本身就是一种只考没见过的题的门槛。
假说 · 尚未验证
作者的诠释(尚未验证): 在作者的课程中,每科题库的题数至少应是允许考试次数的 20 倍(即每次 5 道题的四倍),补考只抽未见过的题,并对 15—16 分的边界卷做人工复核。什么会证明这一判断错:若在真实考试记录中,同一考生补考时遇到旧题与新题的得分差别很小(说明记题不起作用),那么题库规模的要求可以放宽。
小结
5 道题、16 分及格的考试,在掌握度 0.80 附近分辨力很弱:人群中近一半的及格者低于 0.80,约五分之一的胜任者当天不及格。允许补考时,小题库会让不学习的弱考生靠记题通过。考试是过滤器,但这件过滤器的孔径比名义上的「80%」宽得多。
参考文献
Fan, Y. (2026). Diagnostic-to-Pedagogy Conversion Funnels, Mastery-Gated Curriculum Architecture and Certification Network Economics: Self-Cannibalization Bounds, Spaced-Retrieval Curves and Two-Sided Practitioner Licensing [诊学一体转化漏斗、掌握度门控课程与持牌人网络经济学] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090197 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).