What does Active Learning for Intent Labels conclude, in numbers, about “Thousands versus Hundreds: the Counterintuitive Ledger of Uncertainty Sampling”?
Answer: Chapter 1 recomputes the comparison on synthetic customer data with four intents: to reach the target accuracy random sampling needs a median of 1,600 labels and margin sampling 640, a ratio of 0.400; the model trained on every label scores 0.893, and the ratio changes as the target nears the ceiling.
Scope of the numbers: Intent priors, word probabilities and costs are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (23 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.400, 0.893, 1600, 640
| Full pool of 6,000 labels: test accuracy 0.893 | ||
| labels | random(mean of 3) | margin(mean of 3) |
| 80 | 0.768 | 0.810 |
| 160 | 0.815 | 0.840 |
| 320 | 0.851 | 0.867 |
| 640 | 0.868 | 0.882 |
| 1280 | 0.880 | 0.886 |
| 1920 | 0.883 | 0.887 |
| 2560 | 0.886 | 0.889 |
| 3200 | 0.883 | 0.889 |
| labels to reach 0.880, seeds 11 12 13: | ||
| random 1280 1920 1600 | ||
| margin 800 480 640 | ||
| median: random 1,600 vs margin 640; margin uses 0.400 of it |
Show the full program
# Chapter 1. p12: learning curves, random versus margin sampling.
# 合成客服意图数据(固定种子):池 6,000 条、测试 3,000 条、4 个意图。
# 模型:多类逻辑回归,随机梯度下降,每轮从零重训;参数全部为示意。
# 两种策略各跑 3 个种子,记录首次达到目标准确率 0.880 所需标注数。
import math
import random
WORDS = (
"where track tracking ship shipped late delay courier parcel arrive "
"refund money back charge charged return cancel paid reimburse "
"credit size swap small large color exchange fit tight wrong "
"replace agent human person manager complain angry supervisor "
"call speak rude order please hi hello thanks item today help "
"not my the is it can you want").split()
INTENTS = ("track", "refund", "exchange", "human")
V, K = len(WORDS), 4
PRIOR = (0.62, 0.22, 0.11, 0.05) # 意图先验(示意)
PS, PC = 0.40, 0.10 # 本意图词、易混意图词的概率(示意)
CONF = (1, 0, 1, 1) # 每个意图最易混的另一意图
ZW = [1.0 / (r + 1) for r in range(10)] # 意图词内部的齐普夫权重
def make_doc(rng, k):
# 一条去标识化客服消息:3 到 8 个词,表示为(词号, 次数)
cnt = {}
for _ in range(rng.randint(3, 8)):
u = rng.random()
if u < PS:
j = rng.choices(range(10 * k, 10 * k + 10), ZW)[0]
elif u < PS + PC:
c = CONF[k]
j = rng.choices(range(10 * c, 10 * c + 10), ZW)[0]
else:
j = rng.randrange(40, V)
cnt[j] = cnt.get(j, 0) + 1
return tuple(sorted(cnt.items()))
def make_data(seed, n):
rng = random.Random(seed)
xs, ys = [], []
for _ in range(n):
k = rng.choices(range(K), PRIOR)[0]
xs.append(make_doc(rng, k))
ys.append(k)
return xs, ys
def probs(W, x):
# 多类逻辑回归(softmax):每类得分 = 偏置 + 词权重乘计数
s = [math.fsum([W[k][V]] + [W[k][j] * c for j, c in x])
for k in range(K)]
m = max(s)
e = [math.exp(v - m) for v in s]
z = math.fsum(e)
return [v / z for v in e]
def sgd(W, xs, ys, idx, epochs, lr0, rng, lam=0.002):
# 随机梯度下降:每步只更新一次权重(单步状态更新)
order = list(idx)
for ep in range(epochs):
lr = lr0 / (1 + ep)
rng.shuffle(order)
for i in order:
p = probs(W, xs[i])
for k in range(K):
g = p[k] - (1.0 if ys[i] == k else 0.0)
row = W[k]
for j, c in xs[i]:
row[j] -= lr * (g * c + lam * row[j])
row[V] -= lr * g
return W
def fit(xs, ys, idx):
W = [[0.0] * (V + 1) for _ in range(K)]
return sgd(W, xs, ys, idx, 10, 0.3, random.Random(7))
def argmax(p):
return p.index(max(p))
def accuracy(W, xs, ys):
hit = 0
for x, y in zip(xs, ys):
hit += argmax(probs(W, x)) == y
return hit / len(xs)
def score(strategy, p):
# 越小越优先标注
a = sorted(p, reverse=True)
if strategy == "margin":
return a[0] - a[1]
if strategy == "least":
return a[0]
if strategy == "entropy":
return math.fsum(v * math.log(v) for v in p if v > 0)
return -a[0] # "easy": 最有把握的先标
GRID = (20, 30, 40, 60, 80, 100, 120, 160, 200, 240, 320, 400, 480,
640, 800, 960, 1280, 1600, 1920, 2560, 3200)
def run(strategy, seed, pool, test, top):
# 返回 {标注数: 测试准确率} 与已标注下标(按标注先后)
px, py = pool
rng = random.Random(seed)
order = list(range(len(px)))
rng.shuffle(order)
lab = order[:GRID[0]]
taken = set(lab)
W = fit(px, py, lab)
curve = {GRID[0]: accuracy(W, *test)}
for g in GRID[1:]:
if g > top:
break
need = g - len(lab)
if strategy == "random":
cand = order[len(lab):g]
else:
sc = sorted((score(strategy, probs(W, px[i])), i)
for i in range(len(px)) if i not in taken)
cand = [i for _, i in sc[:need]]
taken.update(cand)
lab = lab + cand
W = fit(px, py, lab)
curve[g] = accuracy(W, *test)
return curve, lab, W
def first_reach(curve, target):
for g in sorted(curve):
if curve[g] >= target:
return g
return None
pool, test = make_data(1, 6000), make_data(2, 3000)
full = accuracy(fit(pool[0], pool[1], range(6000)), *test)
SEEDS = (11, 12, 13)
res = {s: [run(s, sd, pool, test, 3200)[0] for sd in SEEDS]
for s in ("random", "margin")}
print("Full pool of 6,000 labels: test accuracy {:.3f}".format(full))
print("labels random(mean of 3) margin(mean of 3)")
for g in (80, 160, 320, 640, 1280, 1920, 2560, 3200):
r = math.fsum(c[g] for c in res["random"]) / 3
m = math.fsum(c[g] for c in res["margin"]) / 3
print("{:6d} {:.3f} {:.3f}".format(g, r, m))
T = 0.880
need = {s: [first_reach(c, T) for c in res[s]] for s in res}
print("labels to reach {:.3f}, seeds 11 12 13:".format(T))
for s in ("random", "margin"):
print(" {:<6s} {}".format(s, " ".join(str(g) for g in need[s])))
mr = sorted(need["random"])[1]
mm = sorted(need["margin"])[1]
print("median: random {:,} vs margin {}; margin uses {:.3f} of it"
.format(mr, mm, mm / mr))Single source and citation
Fan, Y. (2026). Active Learning and Uncertainty Sampling: the Fewest Human Labels a Customer-Intent Classifier Needs to Reach Its Target (Version 1.0.0), Chapter 1: Thousands versus Hundreds: the Counterintuitive Ledger of Uncertainty Sampling. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-312 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026312,
author = {Fan, Yafu},
title = {Active Learning and Uncertainty Sampling: the Fewest Human Labels a Customer-Intent Classifier Needs to Reach Its Target},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 2 · A Hand-Written Multinomial Logistic Regression: Stochastic Gradient Descent on Integer Bag-of-Words Features
- Chapter 3 · Three Rulers of Doubt: Least Confidence, Smallest Margin and Entropy
- Chapter 4 · Sampling Bias: Picked Messages No Longer Stand for the Traffic
- Chapter 5 · Importance Weighting and Prior Adjustment: Computing the Bias Away
- Chapter 6 · Labeling Budgets, Stopping Rules and Audit Sample Sizes
- Chapter 7 · Eastern and Western Classics: the Xueji's Easy-First Order and Bacon's Crucial Instances
- Chapter 8 · Closing the Loop: a Labeling Queue for Customer Intents and a Plan to Refute the Claim