What does Active Learning for Intent Labels conclude, in numbers, about “A Hand-Written Multinomial Logistic Regression: Stochastic Gradient Descent on Integer Bag-of-Words Features”?
Answer: A hand-written logistic regression together with synthetic data whose ceiling can be computed exactly gives a clean test bench: the gradient has passed a numerical check, the training path is repeatable, the ceiling is 0.891, and the model trained on every label already touches it. We can now return to the central question of active learning. Of the four probabilities the model gives, which kind of doubt is most worth sending to a person? Will least confidence, smallest margin and entropy pick the same messages?
Scope of the numbers: Intent priors, word probabilities and costs are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (23 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.891
| check: word probabilities sum to 1.000000 for each intent | |
| intent test n Bayes recall model recall (6,000 labels) | |
| track | 1856 0.921 0.933 |
| refund | 671 0.776 0.753 |
| exchange | 324 0.929 0.929 |
| human | 149 0.940 0.940 |
| overall accuracy: Bayes 0.891, model 0.893 | |
| gap to Bayes ceiling | -0.002 |
| standard error of a 3,000-item accuracy | 0.0057 |
| targets above 0.879 lie within two SE of the ceiling |
Show the full program
# Chapter 2. p23: the ceiling, Bayes accuracy versus the fitted model.
# 数据由已知的生成过程产生,故可写出精确的贝叶斯后验:
# P(k | x) 正比于 先验_k 乘以 每个词在意图 k 下的概率的计数次幂。
# 任何分类器的期望准确率都不会超过贝叶斯准确率(示意数据)。
import math
import random
WORDS = (
"where track tracking ship shipped late delay courier parcel arrive "
"refund money back charge charged return cancel paid reimburse "
"credit size swap small large color exchange fit tight wrong "
"replace agent human person manager complain angry supervisor "
"call speak rude order please hi hello thanks item today help "
"not my the is it can you want").split()
INTENTS = ("track", "refund", "exchange", "human")
V, K = len(WORDS), 4
PRIOR = (0.62, 0.22, 0.11, 0.05) # 意图先验(示意)
PS, PC = 0.40, 0.10 # 本意图词、易混意图词的概率(示意)
CONF = (1, 0, 1, 1) # 每个意图最易混的另一意图
ZW = [1.0 / (r + 1) for r in range(10)] # 意图词内部的齐普夫权重
def make_doc(rng, k):
# 一条去标识化客服消息:3 到 8 个词,表示为(词号, 次数)
cnt = {}
for _ in range(rng.randint(3, 8)):
u = rng.random()
if u < PS:
j = rng.choices(range(10 * k, 10 * k + 10), ZW)[0]
elif u < PS + PC:
c = CONF[k]
j = rng.choices(range(10 * c, 10 * c + 10), ZW)[0]
else:
j = rng.randrange(40, V)
cnt[j] = cnt.get(j, 0) + 1
return tuple(sorted(cnt.items()))
def make_data(seed, n):
rng = random.Random(seed)
xs, ys = [], []
for _ in range(n):
k = rng.choices(range(K), PRIOR)[0]
xs.append(make_doc(rng, k))
ys.append(k)
return xs, ys
def probs(W, x):
# 多类逻辑回归(softmax):每类得分 = 偏置 + 词权重乘计数
s = [math.fsum([W[k][V]] + [W[k][j] * c for j, c in x])
for k in range(K)]
m = max(s)
e = [math.exp(v - m) for v in s]
z = math.fsum(e)
return [v / z for v in e]
def sgd(W, xs, ys, idx, epochs, lr0, rng, lam=0.002):
# 随机梯度下降:每步只更新一次权重(单步状态更新)
order = list(idx)
for ep in range(epochs):
lr = lr0 / (1 + ep)
rng.shuffle(order)
for i in order:
p = probs(W, xs[i])
for k in range(K):
g = p[k] - (1.0 if ys[i] == k else 0.0)
row = W[k]
for j, c in xs[i]:
row[j] -= lr * (g * c + lam * row[j])
row[V] -= lr * g
return W
def fit(xs, ys, idx):
W = [[0.0] * (V + 1) for _ in range(K)]
return sgd(W, xs, ys, idx, 10, 0.3, random.Random(7))
def argmax(p):
return p.index(max(p))
def accuracy(W, xs, ys):
hit = 0
for x, y in zip(xs, ys):
hit += argmax(probs(W, x)) == y
return hit / len(xs)
def word_prob(k):
z = math.fsum(ZW)
pw = [(1 - PS - PC) / (V - 40) if j >= 40 else 0.0 for j in range(V)]
for r in range(10):
pw[10 * k + r] = PS * ZW[r] / z # 本意图词与易混意图词
pw[10 * CONF[k] + r] = PC * ZW[r] / z # 两组词互不重叠
return pw
PW = [word_prob(k) for k in range(K)]
def bayes(x):
s = []
for k in range(K):
if any(PW[k][j] == 0.0 for j, _ in x):
s.append(-math.inf) # 该意图不可能产生此消息
continue
s.append(math.log(PRIOR[k]) + math.fsum(
c * math.log(PW[k][j]) for j, c in x))
return s.index(max(s))
pool, test = make_data(1, 6000), make_data(2, 3000)
tx, ty = test
W = fit(pool[0], pool[1], range(6000))
print("check: word probabilities sum to {:.6f} for each intent".format(
min(math.fsum(PW[k]) for k in range(K))))
print("intent test n Bayes recall model recall (6,000 labels)")
for k in range(K):
idx = [i for i in range(len(ty)) if ty[i] == k]
b = [bayes(tx[i]) == k for i in idx].count(True) / len(idx)
m = [argmax(probs(W, tx[i])) == k for i in idx].count(True) / len(idx)
print("{:<9s} {:5d} {:.3f} {:.3f}".format(
INTENTS[k], len(idx), b, m))
ba = [bayes(x) == y for x, y in zip(tx, ty)].count(True) / len(ty)
print("overall accuracy: Bayes {:.3f}, model {:.3f}".format(
ba, accuracy(W, *test)))
print("gap to Bayes ceiling: {:.3f}".format(ba - accuracy(W, *test)))
se = math.sqrt(ba * (1 - ba) / len(ty))
print("standard error of a {:,}-item accuracy: {:.4f}".format(len(ty), se))
print("targets above {:.3f} lie within two SE of the ceiling".format(
ba - 2 * se))Single source and citation
Fan, Y. (2026). Active Learning and Uncertainty Sampling: the Fewest Human Labels a Customer-Intent Classifier Needs to Reach Its Target (Version 1.0.0), Chapter 2: A Hand-Written Multinomial Logistic Regression: Stochastic Gradient Descent on Integer Bag-of-Words Features. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-312 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026312,
author = {Fan, Yafu},
title = {Active Learning and Uncertainty Sampling: the Fewest Human Labels a Customer-Intent Classifier Needs to Reach Its Target},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Thousands versus Hundreds: the Counterintuitive Ledger of Uncertainty Sampling
- Chapter 3 · Three Rulers of Doubt: Least Confidence, Smallest Margin and Entropy
- Chapter 4 · Sampling Bias: Picked Messages No Longer Stand for the Traffic
- Chapter 5 · Importance Weighting and Prior Adjustment: Computing the Bias Away
- Chapter 6 · Labeling Budgets, Stopping Rules and Audit Sample Sizes
- Chapter 7 · Eastern and Western Classics: the Xueji's Easy-First Order and Bacon's Crucial Instances
- Chapter 8 · Closing the Loop: a Labeling Queue for Customer Intents and a Plan to Refute the Claim