What does Active Learning for Intent Labels conclude, in numbers, about “Importance Weighting and Prior Adjustment: Computing the Bias Away”?
Answer: Chapter 5 draws with known inclusion probabilities and weights by their inverse, after Horvitz and Thompson, bringing the grade of a model whose true accuracy is 0.809 back from the plain mean of 0.657 to the truth, and shows that weighted training and prior-only adjustment can do worse in finite samples.
Scope of the numbers: Intent priors, word probabilities and costs are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (23 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.657, 0.809
grading the 200-label model, naive: mean 0.657, SD 0.016 grading the 200-label model, Hajek: mean 0.809, SD 0.019 its true accuracy on the 5,800 unlabeled: 0.809 one draw: 826 labels in total (200 random seed + 626 sampled) unweighted accuracy 0.875 ECE 0.006 prior TV 0.007 weighted accuracy 0.852 ECE 0.048 prior TV 0.022
Show the full program
# Chapter 5. p52: weighted grading and weighted training.
# 同 p51 的抽样设计(eps = 0.1,预算 600)。其一:用抽到的样本给一个
# 固定模型打分,直接平均 vs Hajek 加权 vs 真值(300 次重复)。
# 其二:用一次抽样训练模型,样本权重 1/pi 归一到均值 1、截断于 10,
# 比较不加权与加权的测试准确率、ECE 与预测先验偏差(示意数据)。
import math
import random
WORDS = (
"where track tracking ship shipped late delay courier parcel arrive "
"refund money back charge charged return cancel paid reimburse "
"credit size swap small large color exchange fit tight wrong "
"replace agent human person manager complain angry supervisor "
"call speak rude order please hi hello thanks item today help "
"not my the is it can you want").split()
INTENTS = ("track", "refund", "exchange", "human")
V, K = len(WORDS), 4
PRIOR = (0.62, 0.22, 0.11, 0.05) # 意图先验(示意)
PS, PC = 0.40, 0.10 # 本意图词、易混意图词的概率(示意)
CONF = (1, 0, 1, 1) # 每个意图最易混的另一意图
ZW = [1.0 / (r + 1) for r in range(10)] # 意图词内部的齐普夫权重
def make_doc(rng, k):
# 一条去标识化客服消息:3 到 8 个词,表示为(词号, 次数)
cnt = {}
for _ in range(rng.randint(3, 8)):
u = rng.random()
if u < PS:
j = rng.choices(range(10 * k, 10 * k + 10), ZW)[0]
elif u < PS + PC:
c = CONF[k]
j = rng.choices(range(10 * c, 10 * c + 10), ZW)[0]
else:
j = rng.randrange(40, V)
cnt[j] = cnt.get(j, 0) + 1
return tuple(sorted(cnt.items()))
def make_data(seed, n):
rng = random.Random(seed)
xs, ys = [], []
for _ in range(n):
k = rng.choices(range(K), PRIOR)[0]
xs.append(make_doc(rng, k))
ys.append(k)
return xs, ys
def probs(W, x):
# 多类逻辑回归(softmax):每类得分 = 偏置 + 词权重乘计数
s = [math.fsum([W[k][V]] + [W[k][j] * c for j, c in x])
for k in range(K)]
m = max(s)
e = [math.exp(v - m) for v in s]
z = math.fsum(e)
return [v / z for v in e]
def sgd(W, xs, ys, idx, epochs, lr0, rng, lam=0.002):
# 随机梯度下降:每步只更新一次权重(单步状态更新)
order = list(idx)
for ep in range(epochs):
lr = lr0 / (1 + ep)
rng.shuffle(order)
for i in order:
p = probs(W, xs[i])
for k in range(K):
g = p[k] - (1.0 if ys[i] == k else 0.0)
row = W[k]
for j, c in xs[i]:
row[j] -= lr * (g * c + lam * row[j])
row[V] -= lr * g
return W
def fit(xs, ys, idx):
W = [[0.0] * (V + 1) for _ in range(K)]
return sgd(W, xs, ys, idx, 10, 0.3, random.Random(7))
def argmax(p):
return p.index(max(p))
def accuracy(W, xs, ys):
hit = 0
for x, y in zip(xs, ys):
hit += argmax(probs(W, x)) == y
return hit / len(xs)
def wfit(xs, ys, idx, wt):
W = [[0.0] * (V + 1) for _ in range(K)]
rng = random.Random(7)
order = list(range(len(idx)))
for ep in range(10):
lr = 0.3 / (1 + ep)
rng.shuffle(order)
for t in order:
i = idx[t]
p = probs(W, xs[i])
for k in range(K):
g = wt[t] * (p[k] - (1.0 if ys[i] == k else 0.0))
for j, c in xs[i]:
W[k][j] -= lr * (g * c + 0.002 * W[k][j])
W[k][V] -= lr * g
return W
def ece(W, xs, ys):
cells = [[] for _ in range(10)]
for x, y in zip(xs, ys):
p = probs(W, x)
cells[min(9, int(max(p) * 10))].append((max(p), argmax(p) == y))
gaps = [len(c) * abs(math.fsum(v for v, _ in c) / len(c)
- [h for _, h in c].count(True) / len(c))
for c in cells if c]
return math.fsum(gaps) / len(xs)
pool, test = make_data(1, 6000), make_data(2, 3000)
px, py = pool
rng = random.Random(11)
order = list(range(6000))
rng.shuffle(order)
W0 = fit(px, py, order[:200])
rest = order[200:]
N = len(rest)
u = []
for i in rest:
a = sorted(probs(W0, px[i]), reverse=True)
u.append(1 - (a[0] - a[1]))
U = math.fsum(u)
pi = [min(1.0, 600 * (0.1 / N + 0.9 * v / U)) for v in u]
hit = [argmax(probs(W0, px[i])) == py[i] for i in rest]
true_acc = hit.count(True) / N
draw = random.Random(20261113)
nv, hj = [], []
for _ in range(300):
S = [t for t in range(N) if draw.random() < pi[t]]
nv.append([hit[t] for t in S].count(True) / len(S))
w = [1 / pi[t] for t in S]
hj.append(math.fsum(w[a] for a, t in enumerate(S) if hit[t])
/ math.fsum(w))
for name, e in (("naive", nv), ("Hajek", hj)):
m = math.fsum(e) / len(e)
sd = math.sqrt(math.fsum((v - m) ** 2 for v in e) / (len(e) - 1))
print("grading the 200-label model, {}: mean {:.3f}, SD {:.3f}"
.format(name, m, sd))
print("its true accuracy on the {:,} unlabeled: {:.3f}".format(
N, true_acc))
S = [t for t in range(N) if draw.random() < pi[t]]
idx = order[:200] + [rest[t] for t in S]
raw = [1.0] * 200 + [1 / pi[t] for t in S]
mean_w = math.fsum(raw) / len(raw)
wt = [min(10.0, v / mean_w) for v in raw]
tshare = [test[1].count(k) / len(test[1]) for k in range(K)]
print("one draw: {} labels in total ({} random seed + {} sampled)".format(
len(idx), 200, len(S)))
for name, ww in (("unweighted", [1.0] * len(idx)), ("weighted", wt)):
W = wfit(px, py, idx, ww)
P = [probs(W, x) for x in test[0]]
mp = [math.fsum(p[k] for p in P) / len(P) for k in range(K)]
tv = 0.5 * math.fsum(abs(a - b) for a, b in zip(mp, tshare))
print(" {:<10s} accuracy {:.3f} ECE {:.3f} prior TV {:.3f}".format(
name, accuracy(W, *test), ece(W, *test), tv))Single source and citation
Fan, Y. (2026). Active Learning and Uncertainty Sampling: the Fewest Human Labels a Customer-Intent Classifier Needs to Reach Its Target (Version 1.0.0), Chapter 5: Importance Weighting and Prior Adjustment: Computing the Bias Away. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-312 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026312,
author = {Fan, Yafu},
title = {Active Learning and Uncertainty Sampling: the Fewest Human Labels a Customer-Intent Classifier Needs to Reach Its Target},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Thousands versus Hundreds: the Counterintuitive Ledger of Uncertainty Sampling
- Chapter 2 · A Hand-Written Multinomial Logistic Regression: Stochastic Gradient Descent on Integer Bag-of-Words Features
- Chapter 3 · Three Rulers of Doubt: Least Confidence, Smallest Margin and Entropy
- Chapter 4 · Sampling Bias: Picked Messages No Longer Stand for the Traffic
- Chapter 6 · Labeling Budgets, Stopping Rules and Audit Sample Sizes
- Chapter 7 · Eastern and Western Classics: the Xueji's Easy-First Order and Bacon's Crucial Instances
- Chapter 8 · Closing the Loop: a Labeling Queue for Customer Intents and a Plan to Refute the Claim