What does Active Learning for Intent Labels conclude, in numbers, about “Thousands versus Hundreds: the Counterintuitive Ledger of Uncertainty Sampling”?

Answer: Chapter 1 recomputes the comparison on synthetic customer data with four intents: to reach the target accuracy random sampling needs a median of 1,600 labels and margin sampling 640, a ratio of 0.400; the model trained on every label scores 0.893, and the ratio changes as the target nears the ceiling.

Scope of the numbers: Intent priors, word probabilities and costs are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (23 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.400, 0.893, 1600, 640

Full pool of 6,000 labels: test accuracy 0.893
labelsrandom(mean of 3)margin(mean of 3)
800.7680.810
1600.8150.840
3200.8510.867
6400.8680.882
12800.8800.886
19200.8830.887
25600.8860.889
32000.8830.889
labels to reach 0.880, seeds 11 12 13:
random 1280 1920 1600
margin 800 480 640
median: random 1,600 vs margin 640; margin uses 0.400 of it
Show the full program
# Chapter 1. p12: learning curves, random versus margin sampling.
# 合成客服意图数据(固定种子):池 6,000 条、测试 3,000 条、4 个意图。
# 模型:多类逻辑回归,随机梯度下降,每轮从零重训;参数全部为示意。
# 两种策略各跑 3 个种子,记录首次达到目标准确率 0.880 所需标注数。
import math
import random

WORDS = (
    "where track tracking ship shipped late delay courier parcel arrive "
    "refund money back charge charged return cancel paid reimburse "
    "credit size swap small large color exchange fit tight wrong "
    "replace agent human person manager complain angry supervisor "
    "call speak rude order please hi hello thanks item today help "
    "not my the is it can you want").split()
INTENTS = ("track", "refund", "exchange", "human")
V, K = len(WORDS), 4
PRIOR = (0.62, 0.22, 0.11, 0.05)   # 意图先验(示意)
PS, PC = 0.40, 0.10                # 本意图词、易混意图词的概率(示意)
CONF = (1, 0, 1, 1)                # 每个意图最易混的另一意图
ZW = [1.0 / (r + 1) for r in range(10)]   # 意图词内部的齐普夫权重


def make_doc(rng, k):
    # 一条去标识化客服消息:3 到 8 个词,表示为(词号, 次数)
    cnt = {}
    for _ in range(rng.randint(3, 8)):
        u = rng.random()
        if u < PS:
            j = rng.choices(range(10 * k, 10 * k + 10), ZW)[0]
        elif u < PS + PC:
            c = CONF[k]
            j = rng.choices(range(10 * c, 10 * c + 10), ZW)[0]
        else:
            j = rng.randrange(40, V)
        cnt[j] = cnt.get(j, 0) + 1
    return tuple(sorted(cnt.items()))


def make_data(seed, n):
    rng = random.Random(seed)
    xs, ys = [], []
    for _ in range(n):
        k = rng.choices(range(K), PRIOR)[0]
        xs.append(make_doc(rng, k))
        ys.append(k)
    return xs, ys


def probs(W, x):
    # 多类逻辑回归(softmax):每类得分 = 偏置 + 词权重乘计数
    s = [math.fsum([W[k][V]] + [W[k][j] * c for j, c in x])
         for k in range(K)]
    m = max(s)
    e = [math.exp(v - m) for v in s]
    z = math.fsum(e)
    return [v / z for v in e]


def sgd(W, xs, ys, idx, epochs, lr0, rng, lam=0.002):
    # 随机梯度下降:每步只更新一次权重(单步状态更新)
    order = list(idx)
    for ep in range(epochs):
        lr = lr0 / (1 + ep)
        rng.shuffle(order)
        for i in order:
            p = probs(W, xs[i])
            for k in range(K):
                g = p[k] - (1.0 if ys[i] == k else 0.0)
                row = W[k]
                for j, c in xs[i]:
                    row[j] -= lr * (g * c + lam * row[j])
                row[V] -= lr * g
    return W


def fit(xs, ys, idx):
    W = [[0.0] * (V + 1) for _ in range(K)]
    return sgd(W, xs, ys, idx, 10, 0.3, random.Random(7))


def argmax(p):
    return p.index(max(p))


def accuracy(W, xs, ys):
    hit = 0
    for x, y in zip(xs, ys):
        hit += argmax(probs(W, x)) == y
    return hit / len(xs)


def score(strategy, p):
    # 越小越优先标注
    a = sorted(p, reverse=True)
    if strategy == "margin":
        return a[0] - a[1]
    if strategy == "least":
        return a[0]
    if strategy == "entropy":
        return math.fsum(v * math.log(v) for v in p if v > 0)
    return -a[0]                     # "easy": 最有把握的先标


GRID = (20, 30, 40, 60, 80, 100, 120, 160, 200, 240, 320, 400, 480,
        640, 800, 960, 1280, 1600, 1920, 2560, 3200)


def run(strategy, seed, pool, test, top):
    # 返回 {标注数: 测试准确率} 与已标注下标(按标注先后)
    px, py = pool
    rng = random.Random(seed)
    order = list(range(len(px)))
    rng.shuffle(order)
    lab = order[:GRID[0]]
    taken = set(lab)
    W = fit(px, py, lab)
    curve = {GRID[0]: accuracy(W, *test)}
    for g in GRID[1:]:
        if g > top:
            break
        need = g - len(lab)
        if strategy == "random":
            cand = order[len(lab):g]
        else:
            sc = sorted((score(strategy, probs(W, px[i])), i)
                        for i in range(len(px)) if i not in taken)
            cand = [i for _, i in sc[:need]]
        taken.update(cand)
        lab = lab + cand
        W = fit(px, py, lab)
        curve[g] = accuracy(W, *test)
    return curve, lab, W


def first_reach(curve, target):
    for g in sorted(curve):
        if curve[g] >= target:
            return g
    return None


pool, test = make_data(1, 6000), make_data(2, 3000)
full = accuracy(fit(pool[0], pool[1], range(6000)), *test)
SEEDS = (11, 12, 13)
res = {s: [run(s, sd, pool, test, 3200)[0] for sd in SEEDS]
       for s in ("random", "margin")}
print("Full pool of 6,000 labels: test accuracy {:.3f}".format(full))
print("labels   random(mean of 3)   margin(mean of 3)")
for g in (80, 160, 320, 640, 1280, 1920, 2560, 3200):
    r = math.fsum(c[g] for c in res["random"]) / 3
    m = math.fsum(c[g] for c in res["margin"]) / 3
    print("{:6d}        {:.3f}               {:.3f}".format(g, r, m))
T = 0.880
need = {s: [first_reach(c, T) for c in res[s]] for s in res}
print("labels to reach {:.3f}, seeds 11 12 13:".format(T))
for s in ("random", "margin"):
    print("  {:<6s} {}".format(s, " ".join(str(g) for g in need[s])))
mr = sorted(need["random"])[1]
mm = sorted(need["margin"])[1]
print("median: random {:,} vs margin {}; margin uses {:.3f} of it"
      .format(mr, mm, mm / mr))

Single source and citation

Fan, Y. (2026). Active Learning and Uncertainty Sampling: the Fewest Human Labels a Customer-Intent Classifier Needs to Reach Its Target (Version 1.0.0), Chapter 1: Thousands versus Hundreds: the Counterintuitive Ledger of Uncertainty Sampling. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-312 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026312,
  author    = {Fan, Yafu},
  title     = {Active Learning and Uncertainty Sampling: the Fewest Human Labels a Customer-Intent Classifier Needs to Reach Its Target},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book