关于「三把尺子量「拿不准」:最小置信、最小间隔与熵」,《主动学习与客服意图标注》给出的定量结论是什么?
结论:在本书的数据上,第一个隐患不严重:最小间隔一次挑出的 200 条里有 197 条互不相同,整个池子 6,000 条中完全重复的消息有 133 条。但这一批的构成与随机批截然不同:随机批里查物流 127 条、退款 40 条、转人工 10 条;最小间隔批里查物流只有 83 条,退款 69 条,转人工 26 条,消息平均也更短,为 5.17 个词,随机批是 5.85 个。最小间隔把标注员的时间集中在退款与查物流的交界和稀有意图上,这正是它省标注的原因,也正是它带来偏差的原因。
数据口径:书中意图先验、词概率与成本均为示意参数;全部计算由书中程序生成;本书属于非个人化经营建议。 第一章在合成的四意图客服数据上重算:达到目标准确率,随机抽样的中位数是 1,600 条,最小间隔采样是 640 条,后者只用前者的 0.400;全部标注训练的模型准确率为 0.893,这一倍数随目标逼近天花板而变化。
对照数据表
下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 10, 127, 133, 197, 200, 26
| margin batch of 200 | 197 distinct messages, mean length 5.17 |
| true intents in batch: track | 83 refund=69 exchange=22 human=26 |
| random batch of 200 | 200 distinct messages, mean length 5.85 |
| true intents in batch: track | 127 refund=40 exchange=23 human=10 |
| exact duplicates in the whole pool | 133 of 6,000 |
| cold start: P(seed of m random messages has no 'human') | |
| m | 10: 0.5987 |
| m | 20: 0.3585 |
| m | 40: 0.1285 |
| m | 80: 0.0165 |
| m | 160: 0.0003 |
| smallest m with P below 0.05 | 59 |
查看完整验算程序
# Chapter 3. p33: batch redundancy and the cold start.
# 一次挑 200 条最拿不准的:重复多不多、短消息多不多、意图构成如何?
# 冷启动:随机种子集 m 条全都不含先验 0.05 的意图的精确概率 0.95^m。
import math
import random
from fractions import Fraction
WORDS = (
"where track tracking ship shipped late delay courier parcel arrive "
"refund money back charge charged return cancel paid reimburse "
"credit size swap small large color exchange fit tight wrong "
"replace agent human person manager complain angry supervisor "
"call speak rude order please hi hello thanks item today help "
"not my the is it can you want").split()
INTENTS = ("track", "refund", "exchange", "human")
V, K = len(WORDS), 4
PRIOR = (0.62, 0.22, 0.11, 0.05) # 意图先验(示意)
PS, PC = 0.40, 0.10 # 本意图词、易混意图词的概率(示意)
CONF = (1, 0, 1, 1) # 每个意图最易混的另一意图
ZW = [1.0 / (r + 1) for r in range(10)] # 意图词内部的齐普夫权重
def make_doc(rng, k):
# 一条去标识化客服消息:3 到 8 个词,表示为(词号, 次数)
cnt = {}
for _ in range(rng.randint(3, 8)):
u = rng.random()
if u < PS:
j = rng.choices(range(10 * k, 10 * k + 10), ZW)[0]
elif u < PS + PC:
c = CONF[k]
j = rng.choices(range(10 * c, 10 * c + 10), ZW)[0]
else:
j = rng.randrange(40, V)
cnt[j] = cnt.get(j, 0) + 1
return tuple(sorted(cnt.items()))
def make_data(seed, n):
rng = random.Random(seed)
xs, ys = [], []
for _ in range(n):
k = rng.choices(range(K), PRIOR)[0]
xs.append(make_doc(rng, k))
ys.append(k)
return xs, ys
def probs(W, x):
# 多类逻辑回归(softmax):每类得分 = 偏置 + 词权重乘计数
s = [math.fsum([W[k][V]] + [W[k][j] * c for j, c in x])
for k in range(K)]
m = max(s)
e = [math.exp(v - m) for v in s]
z = math.fsum(e)
return [v / z for v in e]
def sgd(W, xs, ys, idx, epochs, lr0, rng, lam=0.002):
# 随机梯度下降:每步只更新一次权重(单步状态更新)
order = list(idx)
for ep in range(epochs):
lr = lr0 / (1 + ep)
rng.shuffle(order)
for i in order:
p = probs(W, xs[i])
for k in range(K):
g = p[k] - (1.0 if ys[i] == k else 0.0)
row = W[k]
for j, c in xs[i]:
row[j] -= lr * (g * c + lam * row[j])
row[V] -= lr * g
return W
def fit(xs, ys, idx):
W = [[0.0] * (V + 1) for _ in range(K)]
return sgd(W, xs, ys, idx, 10, 0.3, random.Random(7))
def argmax(p):
return p.index(max(p))
def accuracy(W, xs, ys):
hit = 0
for x, y in zip(xs, ys):
hit += argmax(probs(W, x)) == y
return hit / len(xs)
def score(strategy, p):
# 越小越优先标注
a = sorted(p, reverse=True)
if strategy == "margin":
return a[0] - a[1]
if strategy == "least":
return a[0]
if strategy == "entropy":
return math.fsum(v * math.log(v) for v in p if v > 0)
return -a[0] # "easy": 最有把握的先标
GRID = (20, 30, 40, 60, 80, 100, 120, 160, 200, 240, 320, 400, 480,
640, 800, 960, 1280, 1600, 1920, 2560, 3200)
def run(strategy, seed, pool, test, top):
# 返回 {标注数: 测试准确率} 与已标注下标(按标注先后)
px, py = pool
rng = random.Random(seed)
order = list(range(len(px)))
rng.shuffle(order)
lab = order[:GRID[0]]
taken = set(lab)
W = fit(px, py, lab)
curve = {GRID[0]: accuracy(W, *test)}
for g in GRID[1:]:
if g > top:
break
need = g - len(lab)
if strategy == "random":
cand = order[len(lab):g]
else:
sc = sorted((score(strategy, probs(W, px[i])), i)
for i in range(len(px)) if i not in taken)
cand = [i for _, i in sc[:need]]
taken.update(cand)
lab = lab + cand
W = fit(px, py, lab)
curve[g] = accuracy(W, *test)
return curve, lab, W
def first_reach(curve, target):
for g in sorted(curve):
if curve[g] >= target:
return g
return None
pool, test = make_data(1, 6000), make_data(2, 3000)
px, py = pool
rng = random.Random(11)
order = list(range(6000))
rng.shuffle(order)
W = fit(px, py, order[:160])
rest = order[160:]
sc = sorted((score("margin", probs(W, px[i])), i) for i in rest)
batch_m = [i for _, i in sc[:200]]
batch_r = rest[:200]
for name, bt in (("margin", batch_m), ("random", batch_r)):
distinct = len(set(px[i] for i in bt))
mean_len = math.fsum(c for i in bt for _, c in px[i]) / len(bt)
cls = [[py[i] for i in bt].count(k) for k in range(K)]
print("{:<6s} batch of 200: {:3d} distinct messages, mean length "
"{:.2f}".format(name, distinct, mean_len))
print(" true intents in batch: " + " ".join(
"{}={}".format(INTENTS[k], cls[k]) for k in range(K)))
dup_pool = 6000 - len(set(px))
print("exact duplicates in the whole pool: {:,} of 6,000".format(dup_pool))
print("cold start: P(seed of m random messages has no 'human')")
for m in (10, 20, 40, 80, 160):
p = Fraction(95, 100) ** m
print(" m = {:3d}: {:.4f}".format(m, float(p)))
need = math.ceil(math.log(0.05) / math.log(0.95))
print("smallest m with P below 0.05: {}".format(need))唯一数据源与引用
范雅福(Fan, Y.). (2026). 《主动学习与不确定性采样、最少人工标注让客服意图分类器达标母典》第 3 章「三把尺子量「拿不准」:最小置信、最小间隔与熵」(版本 1.0.0). 5D Library. DOI 待发布
DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-312 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026312,
author = {Fan, Yafu},
title = {Active Learning and Uncertainty Sampling: the Fewest Human Labels a Customer-Intent Classifier Needs to Reach Its Target},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}