关于「手写多类逻辑回归:词袋整数特征上的随机梯度下降」,《主动学习与客服意图标注》给出的定量结论是什么?
结论:一个手写的逻辑回归,加上可以精确算出上限的合成数据,给了我们一个干净的实验台:梯度经过数值检验,训练轨迹可重复,天花板为 0.891,用全部标注的模型已经贴着它。现在可以回到主动学习的核心问题:模型给出的四个概率里,哪一种「拿不准」最值得送去标注?最小置信、最小间隔和熵,挑出来的会是同一批消息吗?
数据口径:书中意图先验、词概率与成本均为示意参数;全部计算由书中程序生成;本书属于非个人化经营建议。 第一章在合成的四意图客服数据上重算:达到目标准确率,随机抽样的中位数是 1,600 条,最小间隔采样是 640 条,后者只用前者的 0.400;全部标注训练的模型准确率为 0.893,这一倍数随目标逼近天花板而变化。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.891
| check: word probabilities sum to 1.000000 for each intent | |
| intent test n Bayes recall model recall (6,000 labels) | |
| track | 1856 0.921 0.933 |
| refund | 671 0.776 0.753 |
| exchange | 324 0.929 0.929 |
| human | 149 0.940 0.940 |
| overall accuracy: Bayes 0.891, model 0.893 | |
| gap to Bayes ceiling | -0.002 |
| standard error of a 3,000-item accuracy | 0.0057 |
| targets above 0.879 lie within two SE of the ceiling |
查看完整验算程序
# Chapter 2. p23: the ceiling, Bayes accuracy versus the fitted model.
# 数据由已知的生成过程产生,故可写出精确的贝叶斯后验:
# P(k | x) 正比于 先验_k 乘以 每个词在意图 k 下的概率的计数次幂。
# 任何分类器的期望准确率都不会超过贝叶斯准确率(示意数据)。
import math
import random
WORDS = (
"where track tracking ship shipped late delay courier parcel arrive "
"refund money back charge charged return cancel paid reimburse "
"credit size swap small large color exchange fit tight wrong "
"replace agent human person manager complain angry supervisor "
"call speak rude order please hi hello thanks item today help "
"not my the is it can you want").split()
INTENTS = ("track", "refund", "exchange", "human")
V, K = len(WORDS), 4
PRIOR = (0.62, 0.22, 0.11, 0.05) # 意图先验(示意)
PS, PC = 0.40, 0.10 # 本意图词、易混意图词的概率(示意)
CONF = (1, 0, 1, 1) # 每个意图最易混的另一意图
ZW = [1.0 / (r + 1) for r in range(10)] # 意图词内部的齐普夫权重
def make_doc(rng, k):
# 一条去标识化客服消息:3 到 8 个词,表示为(词号, 次数)
cnt = {}
for _ in range(rng.randint(3, 8)):
u = rng.random()
if u < PS:
j = rng.choices(range(10 * k, 10 * k + 10), ZW)[0]
elif u < PS + PC:
c = CONF[k]
j = rng.choices(range(10 * c, 10 * c + 10), ZW)[0]
else:
j = rng.randrange(40, V)
cnt[j] = cnt.get(j, 0) + 1
return tuple(sorted(cnt.items()))
def make_data(seed, n):
rng = random.Random(seed)
xs, ys = [], []
for _ in range(n):
k = rng.choices(range(K), PRIOR)[0]
xs.append(make_doc(rng, k))
ys.append(k)
return xs, ys
def probs(W, x):
# 多类逻辑回归(softmax):每类得分 = 偏置 + 词权重乘计数
s = [math.fsum([W[k][V]] + [W[k][j] * c for j, c in x])
for k in range(K)]
m = max(s)
e = [math.exp(v - m) for v in s]
z = math.fsum(e)
return [v / z for v in e]
def sgd(W, xs, ys, idx, epochs, lr0, rng, lam=0.002):
# 随机梯度下降:每步只更新一次权重(单步状态更新)
order = list(idx)
for ep in range(epochs):
lr = lr0 / (1 + ep)
rng.shuffle(order)
for i in order:
p = probs(W, xs[i])
for k in range(K):
g = p[k] - (1.0 if ys[i] == k else 0.0)
row = W[k]
for j, c in xs[i]:
row[j] -= lr * (g * c + lam * row[j])
row[V] -= lr * g
return W
def fit(xs, ys, idx):
W = [[0.0] * (V + 1) for _ in range(K)]
return sgd(W, xs, ys, idx, 10, 0.3, random.Random(7))
def argmax(p):
return p.index(max(p))
def accuracy(W, xs, ys):
hit = 0
for x, y in zip(xs, ys):
hit += argmax(probs(W, x)) == y
return hit / len(xs)
def word_prob(k):
z = math.fsum(ZW)
pw = [(1 - PS - PC) / (V - 40) if j >= 40 else 0.0 for j in range(V)]
for r in range(10):
pw[10 * k + r] = PS * ZW[r] / z # 本意图词与易混意图词
pw[10 * CONF[k] + r] = PC * ZW[r] / z # 两组词互不重叠
return pw
PW = [word_prob(k) for k in range(K)]
def bayes(x):
s = []
for k in range(K):
if any(PW[k][j] == 0.0 for j, _ in x):
s.append(-math.inf) # 该意图不可能产生此消息
continue
s.append(math.log(PRIOR[k]) + math.fsum(
c * math.log(PW[k][j]) for j, c in x))
return s.index(max(s))
pool, test = make_data(1, 6000), make_data(2, 3000)
tx, ty = test
W = fit(pool[0], pool[1], range(6000))
print("check: word probabilities sum to {:.6f} for each intent".format(
min(math.fsum(PW[k]) for k in range(K))))
print("intent test n Bayes recall model recall (6,000 labels)")
for k in range(K):
idx = [i for i in range(len(ty)) if ty[i] == k]
b = [bayes(tx[i]) == k for i in idx].count(True) / len(idx)
m = [argmax(probs(W, tx[i])) == k for i in idx].count(True) / len(idx)
print("{:<9s} {:5d} {:.3f} {:.3f}".format(
INTENTS[k], len(idx), b, m))
ba = [bayes(x) == y for x, y in zip(tx, ty)].count(True) / len(ty)
print("overall accuracy: Bayes {:.3f}, model {:.3f}".format(
ba, accuracy(W, *test)))
print("gap to Bayes ceiling: {:.3f}".format(ba - accuracy(W, *test)))
se = math.sqrt(ba * (1 - ba) / len(ty))
print("standard error of a {:,}-item accuracy: {:.4f}".format(len(ty), se))
print("targets above {:.3f} lie within two SE of the ceiling".format(
ba - 2 * se))唯一数据源与引用
范雅福(Fan, Y.). (2026). 《主动学习与不确定性采样、最少人工标注让客服意图分类器达标母典》第 2 章「手写多类逻辑回归:词袋整数特征上的随机梯度下降」(版本 1.0.0). 5D Library. DOI 待发布
DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-312 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026312,
author = {Fan, Yafu},
title = {Active Learning and Uncertainty Sampling: the Fewest Human Labels a Customer-Intent Classifier Needs to Reach Its Target},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}