What does Active Learning for Intent Labels conclude, in numbers, about “Three Rulers of Doubt: Least Confidence, Smallest Margin and Entropy”?
Answer: On the book's data the first hazard is mild: of the 200 messages one margin batch picks, 197 are distinct, and the whole pool of 6,000 messages contains 133 exact duplicates. Yet the make-up of the batch is quite unlike that of a random one. The random batch holds 127 tracking messages, 40 refunds and 10 hand-overs; the margin batch holds only 83 tracking messages, 69 refunds and 26 hand-overs, and its messages are shorter, 5.17 words on average against 5.85 for the random batch. Margin sampling concentrates the taggers' time on the border between refunds and tracking and on the rare intent, which is why it saves labels and also why it brings bias.
Scope of the numbers: Intent priors, word probabilities and costs are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (23 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 10, 127, 133, 197, 200, 26
| margin batch of 200 | 197 distinct messages, mean length 5.17 |
| true intents in batch: track | 83 refund=69 exchange=22 human=26 |
| random batch of 200 | 200 distinct messages, mean length 5.85 |
| true intents in batch: track | 127 refund=40 exchange=23 human=10 |
| exact duplicates in the whole pool | 133 of 6,000 |
| cold start: P(seed of m random messages has no 'human') | |
| m | 10: 0.5987 |
| m | 20: 0.3585 |
| m | 40: 0.1285 |
| m | 80: 0.0165 |
| m | 160: 0.0003 |
| smallest m with P below 0.05 | 59 |
Show the full program
# Chapter 3. p33: batch redundancy and the cold start.
# 一次挑 200 条最拿不准的:重复多不多、短消息多不多、意图构成如何?
# 冷启动:随机种子集 m 条全都不含先验 0.05 的意图的精确概率 0.95^m。
import math
import random
from fractions import Fraction
WORDS = (
"where track tracking ship shipped late delay courier parcel arrive "
"refund money back charge charged return cancel paid reimburse "
"credit size swap small large color exchange fit tight wrong "
"replace agent human person manager complain angry supervisor "
"call speak rude order please hi hello thanks item today help "
"not my the is it can you want").split()
INTENTS = ("track", "refund", "exchange", "human")
V, K = len(WORDS), 4
PRIOR = (0.62, 0.22, 0.11, 0.05) # 意图先验(示意)
PS, PC = 0.40, 0.10 # 本意图词、易混意图词的概率(示意)
CONF = (1, 0, 1, 1) # 每个意图最易混的另一意图
ZW = [1.0 / (r + 1) for r in range(10)] # 意图词内部的齐普夫权重
def make_doc(rng, k):
# 一条去标识化客服消息:3 到 8 个词,表示为(词号, 次数)
cnt = {}
for _ in range(rng.randint(3, 8)):
u = rng.random()
if u < PS:
j = rng.choices(range(10 * k, 10 * k + 10), ZW)[0]
elif u < PS + PC:
c = CONF[k]
j = rng.choices(range(10 * c, 10 * c + 10), ZW)[0]
else:
j = rng.randrange(40, V)
cnt[j] = cnt.get(j, 0) + 1
return tuple(sorted(cnt.items()))
def make_data(seed, n):
rng = random.Random(seed)
xs, ys = [], []
for _ in range(n):
k = rng.choices(range(K), PRIOR)[0]
xs.append(make_doc(rng, k))
ys.append(k)
return xs, ys
def probs(W, x):
# 多类逻辑回归(softmax):每类得分 = 偏置 + 词权重乘计数
s = [math.fsum([W[k][V]] + [W[k][j] * c for j, c in x])
for k in range(K)]
m = max(s)
e = [math.exp(v - m) for v in s]
z = math.fsum(e)
return [v / z for v in e]
def sgd(W, xs, ys, idx, epochs, lr0, rng, lam=0.002):
# 随机梯度下降:每步只更新一次权重(单步状态更新)
order = list(idx)
for ep in range(epochs):
lr = lr0 / (1 + ep)
rng.shuffle(order)
for i in order:
p = probs(W, xs[i])
for k in range(K):
g = p[k] - (1.0 if ys[i] == k else 0.0)
row = W[k]
for j, c in xs[i]:
row[j] -= lr * (g * c + lam * row[j])
row[V] -= lr * g
return W
def fit(xs, ys, idx):
W = [[0.0] * (V + 1) for _ in range(K)]
return sgd(W, xs, ys, idx, 10, 0.3, random.Random(7))
def argmax(p):
return p.index(max(p))
def accuracy(W, xs, ys):
hit = 0
for x, y in zip(xs, ys):
hit += argmax(probs(W, x)) == y
return hit / len(xs)
def score(strategy, p):
# 越小越优先标注
a = sorted(p, reverse=True)
if strategy == "margin":
return a[0] - a[1]
if strategy == "least":
return a[0]
if strategy == "entropy":
return math.fsum(v * math.log(v) for v in p if v > 0)
return -a[0] # "easy": 最有把握的先标
GRID = (20, 30, 40, 60, 80, 100, 120, 160, 200, 240, 320, 400, 480,
640, 800, 960, 1280, 1600, 1920, 2560, 3200)
def run(strategy, seed, pool, test, top):
# 返回 {标注数: 测试准确率} 与已标注下标(按标注先后)
px, py = pool
rng = random.Random(seed)
order = list(range(len(px)))
rng.shuffle(order)
lab = order[:GRID[0]]
taken = set(lab)
W = fit(px, py, lab)
curve = {GRID[0]: accuracy(W, *test)}
for g in GRID[1:]:
if g > top:
break
need = g - len(lab)
if strategy == "random":
cand = order[len(lab):g]
else:
sc = sorted((score(strategy, probs(W, px[i])), i)
for i in range(len(px)) if i not in taken)
cand = [i for _, i in sc[:need]]
taken.update(cand)
lab = lab + cand
W = fit(px, py, lab)
curve[g] = accuracy(W, *test)
return curve, lab, W
def first_reach(curve, target):
for g in sorted(curve):
if curve[g] >= target:
return g
return None
pool, test = make_data(1, 6000), make_data(2, 3000)
px, py = pool
rng = random.Random(11)
order = list(range(6000))
rng.shuffle(order)
W = fit(px, py, order[:160])
rest = order[160:]
sc = sorted((score("margin", probs(W, px[i])), i) for i in rest)
batch_m = [i for _, i in sc[:200]]
batch_r = rest[:200]
for name, bt in (("margin", batch_m), ("random", batch_r)):
distinct = len(set(px[i] for i in bt))
mean_len = math.fsum(c for i in bt for _, c in px[i]) / len(bt)
cls = [[py[i] for i in bt].count(k) for k in range(K)]
print("{:<6s} batch of 200: {:3d} distinct messages, mean length "
"{:.2f}".format(name, distinct, mean_len))
print(" true intents in batch: " + " ".join(
"{}={}".format(INTENTS[k], cls[k]) for k in range(K)))
dup_pool = 6000 - len(set(px))
print("exact duplicates in the whole pool: {:,} of 6,000".format(dup_pool))
print("cold start: P(seed of m random messages has no 'human')")
for m in (10, 20, 40, 80, 160):
p = Fraction(95, 100) ** m
print(" m = {:3d}: {:.4f}".format(m, float(p)))
need = math.ceil(math.log(0.05) / math.log(0.95))
print("smallest m with P below 0.05: {}".format(need))Single source and citation
Fan, Y. (2026). Active Learning and Uncertainty Sampling: the Fewest Human Labels a Customer-Intent Classifier Needs to Reach Its Target (Version 1.0.0), Chapter 3: Three Rulers of Doubt: Least Confidence, Smallest Margin and Entropy. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-312 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026312,
author = {Fan, Yafu},
title = {Active Learning and Uncertainty Sampling: the Fewest Human Labels a Customer-Intent Classifier Needs to Reach Its Target},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Thousands versus Hundreds: the Counterintuitive Ledger of Uncertainty Sampling
- Chapter 2 · A Hand-Written Multinomial Logistic Regression: Stochastic Gradient Descent on Integer Bag-of-Words Features
- Chapter 4 · Sampling Bias: Picked Messages No Longer Stand for the Traffic
- Chapter 5 · Importance Weighting and Prior Adjustment: Computing the Bias Away
- Chapter 6 · Labeling Budgets, Stopping Rules and Audit Sample Sizes
- Chapter 7 · Eastern and Western Classics: the Xueji's Easy-First Order and Bacon's Crucial Instances
- Chapter 8 · Closing the Loop: a Labeling Queue for Customer Intents and a Plan to Refute the Claim