关于「东西方原典考辨:《学记》的先易后难与培根的关键实例」,《主动学习与客服意图标注》给出的定量结论是什么?
结论:程序 p72 让两个候选模型 A 与 B 各用 160 条互不重叠的随机标注训练,然后问:要判定谁更准,需要标多少条?两个模型只在 0.201 的消息上意见不一;在两者意见一致的消息上,无论标注结果如何,都不能区分它们。在恰有一个判对的 0.184 的消息里,A 判对的比例为 0.432。用双侧检验在 0.05 水平、功效 0.80 下判定这个比例不等于一半,需要 417 条有效实例;只标分歧消息,约需 456 条;随机标注则约需 2,266 条。分歧过滤省下了 0.799 的标注。培根的路标在这里有一个精确的数量含义:它把需要的观察次数缩小到分歧比例分之一左右。
数据口径:书中意图先验、词概率与成本均为示意参数;全部计算由书中程序生成;本书属于非个人化经营建议。 第一章在合成的四意图客服数据上重算:达到目标准确率,随机抽样的中位数是 1,600 条,最小间隔采样是 640 条,后者只用前者的 0.400;全部标注训练的模型准确率为 0.893,这一倍数随目标逼近天花板而变化。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.184, 0.201, 0.432, 0.799, 2266, 417
| accuracy on 5,680 unlabeled: A 0.793 B 0.818 gap -0.025 | |
| A and B disagree on 0.201 of messages | |
| exactly one is right on d | 0.184 of messages |
| of those, A is right on a | 0.432, B on 0.568 |
| informative labels needed | 417 |
| labeling disagreements only: about 456 labels | |
| labeling random messages: about 2,266 labels | |
| share of labels the disagreement filter saves | 0.799 |
查看完整验算程序
# Chapter 7. p72: Bacon's crucial instances as disagreement labels.
# 两个候选模型 A、B 各用 160 条互不重叠的随机标注训练。要判定谁更准,
# 只有两者判断不同、且恰有一个判对的消息才提供信息(「关键实例」)。
# 在这些消息里 A 判对的比例为 a;检验 a = 0.5,双侧 alpha 0.05、
# 功效 0.80,所需有效实例数 m;只标分歧消息约需 m q / d 条(q 为分歧
# 占比),随机抽标约需 m / d 条(d 为恰有一个判对的占比)。
import math
import random
from statistics import NormalDist
WORDS = (
"where track tracking ship shipped late delay courier parcel arrive "
"refund money back charge charged return cancel paid reimburse "
"credit size swap small large color exchange fit tight wrong "
"replace agent human person manager complain angry supervisor "
"call speak rude order please hi hello thanks item today help "
"not my the is it can you want").split()
INTENTS = ("track", "refund", "exchange", "human")
V, K = len(WORDS), 4
PRIOR = (0.62, 0.22, 0.11, 0.05) # 意图先验(示意)
PS, PC = 0.40, 0.10 # 本意图词、易混意图词的概率(示意)
CONF = (1, 0, 1, 1) # 每个意图最易混的另一意图
ZW = [1.0 / (r + 1) for r in range(10)] # 意图词内部的齐普夫权重
def make_doc(rng, k):
# 一条去标识化客服消息:3 到 8 个词,表示为(词号, 次数)
cnt = {}
for _ in range(rng.randint(3, 8)):
u = rng.random()
if u < PS:
j = rng.choices(range(10 * k, 10 * k + 10), ZW)[0]
elif u < PS + PC:
c = CONF[k]
j = rng.choices(range(10 * c, 10 * c + 10), ZW)[0]
else:
j = rng.randrange(40, V)
cnt[j] = cnt.get(j, 0) + 1
return tuple(sorted(cnt.items()))
def make_data(seed, n):
rng = random.Random(seed)
xs, ys = [], []
for _ in range(n):
k = rng.choices(range(K), PRIOR)[0]
xs.append(make_doc(rng, k))
ys.append(k)
return xs, ys
def probs(W, x):
# 多类逻辑回归(softmax):每类得分 = 偏置 + 词权重乘计数
s = [math.fsum([W[k][V]] + [W[k][j] * c for j, c in x])
for k in range(K)]
m = max(s)
e = [math.exp(v - m) for v in s]
z = math.fsum(e)
return [v / z for v in e]
def sgd(W, xs, ys, idx, epochs, lr0, rng, lam=0.002):
# 随机梯度下降:每步只更新一次权重(单步状态更新)
order = list(idx)
for ep in range(epochs):
lr = lr0 / (1 + ep)
rng.shuffle(order)
for i in order:
p = probs(W, xs[i])
for k in range(K):
g = p[k] - (1.0 if ys[i] == k else 0.0)
row = W[k]
for j, c in xs[i]:
row[j] -= lr * (g * c + lam * row[j])
row[V] -= lr * g
return W
def fit(xs, ys, idx):
W = [[0.0] * (V + 1) for _ in range(K)]
return sgd(W, xs, ys, idx, 10, 0.3, random.Random(7))
def argmax(p):
return p.index(max(p))
def accuracy(W, xs, ys):
hit = 0
for x, y in zip(xs, ys):
hit += argmax(probs(W, x)) == y
return hit / len(xs)
pool, test = make_data(1, 6000), make_data(2, 3000)
px, py = pool
rng = random.Random(11)
order = list(range(6000))
rng.shuffle(order)
A = fit(px, py, order[:160])
B = fit(px, py, order[160:320])
rest = order[320:]
pa = [argmax(probs(A, px[i])) for i in rest]
pb = [argmax(probs(B, px[i])) for i in rest]
yy = [py[i] for i in rest]
n = len(rest)
dis = [t for t in range(n) if pa[t] != pb[t]]
only_a = [t for t in dis if pa[t] == yy[t]]
only_b = [t for t in dis if pb[t] == yy[t]]
acc_a = [pa[t] == yy[t] for t in range(n)].count(True) / n
acc_b = [pb[t] == yy[t] for t in range(n)].count(True) / n
d = (len(only_a) + len(only_b)) / n
a = len(only_a) / (len(only_a) + len(only_b))
print("accuracy on {:,} unlabeled: A {:.3f} B {:.3f} gap {:.3f}".format(
n, acc_a, acc_b, acc_a - acc_b))
print("A and B disagree on {:.3f} of messages".format(len(dis) / n))
print("exactly one is right on d = {:.3f} of messages".format(d))
print(" of those, A is right on a = {:.3f}, B on {:.3f}".format(a, 1 - a))
Z = NormalDist()
za, zb = Z.inv_cdf(0.975), Z.inv_cdf(0.80)
m = ((za * 0.5 + zb * math.sqrt(a * (1 - a))) / (a - 0.5)) ** 2
q = len(dis) / n
print("informative labels needed: {}".format(math.ceil(m)))
print("labeling disagreements only: about {} labels".format(
math.ceil(m * q / d)))
print("labeling random messages: about {:,} labels".format(
math.ceil(m / d)))
print("share of labels the disagreement filter saves: {:.3f}".format(
1 - q))唯一数据源与引用
范雅福(Fan, Y.). (2026). 《主动学习与不确定性采样、最少人工标注让客服意图分类器达标母典》第 7 章「东西方原典考辨:《学记》的先易后难与培根的关键实例」(版本 1.0.0). 5D Library. DOI 待发布
DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-312 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026312,
author = {Fan, Yafu},
title = {Active Learning and Uncertainty Sampling: the Fewest Human Labels a Customer-Intent Classifier Needs to Reach Its Target},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}