What does Active Learning for Intent Labels conclude, in numbers, about “Eastern and Western Classics: the Xueji's Easy-First Order and Bacon's Crucial Instances”?

Answer: Program p72 trains two candidate models, A and B, each on 160 random labels that do not overlap, and asks how many labels are needed to decide which is more accurate. The two models disagree on only 0.201 of the messages; where they agree, no label can separate them, whatever it says. Among the 0.184 of messages on which exactly one of them is right, A is right on a share of 0.432. Deciding that this share differs from one half with a two-sided test at the 0.05 level and power 0.80 needs 417 informative instances; labeling only the disagreements needs about 456 labels, and labeling at random about 2,266. The disagreement filter saves 0.799 of the labels. Bacon's fingerpost here acquires a precise quantity: it divides the number of observations needed by roughly the share of disagreement.

Scope of the numbers: Intent priors, word probabilities and costs are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (23 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.184, 0.201, 0.432, 0.799, 2266, 417

accuracy on 5,680 unlabeled: A 0.793 B 0.818 gap -0.025
A and B disagree on 0.201 of messages
exactly one is right on d0.184 of messages
of those, A is right on a0.432, B on 0.568
informative labels needed417
labeling disagreements only: about 456 labels
labeling random messages: about 2,266 labels
share of labels the disagreement filter saves0.799
Show the full program
# Chapter 7. p72: Bacon's crucial instances as disagreement labels.
# 两个候选模型 A、B 各用 160 条互不重叠的随机标注训练。要判定谁更准,
# 只有两者判断不同、且恰有一个判对的消息才提供信息(「关键实例」)。
# 在这些消息里 A 判对的比例为 a;检验 a = 0.5,双侧 alpha 0.05、
# 功效 0.80,所需有效实例数 m;只标分歧消息约需 m q / d 条(q 为分歧
# 占比),随机抽标约需 m / d 条(d 为恰有一个判对的占比)。
import math
import random
from statistics import NormalDist

WORDS = (
    "where track tracking ship shipped late delay courier parcel arrive "
    "refund money back charge charged return cancel paid reimburse "
    "credit size swap small large color exchange fit tight wrong "
    "replace agent human person manager complain angry supervisor "
    "call speak rude order please hi hello thanks item today help "
    "not my the is it can you want").split()
INTENTS = ("track", "refund", "exchange", "human")
V, K = len(WORDS), 4
PRIOR = (0.62, 0.22, 0.11, 0.05)   # 意图先验(示意)
PS, PC = 0.40, 0.10                # 本意图词、易混意图词的概率(示意)
CONF = (1, 0, 1, 1)                # 每个意图最易混的另一意图
ZW = [1.0 / (r + 1) for r in range(10)]   # 意图词内部的齐普夫权重


def make_doc(rng, k):
    # 一条去标识化客服消息:3 到 8 个词,表示为(词号, 次数)
    cnt = {}
    for _ in range(rng.randint(3, 8)):
        u = rng.random()
        if u < PS:
            j = rng.choices(range(10 * k, 10 * k + 10), ZW)[0]
        elif u < PS + PC:
            c = CONF[k]
            j = rng.choices(range(10 * c, 10 * c + 10), ZW)[0]
        else:
            j = rng.randrange(40, V)
        cnt[j] = cnt.get(j, 0) + 1
    return tuple(sorted(cnt.items()))


def make_data(seed, n):
    rng = random.Random(seed)
    xs, ys = [], []
    for _ in range(n):
        k = rng.choices(range(K), PRIOR)[0]
        xs.append(make_doc(rng, k))
        ys.append(k)
    return xs, ys


def probs(W, x):
    # 多类逻辑回归(softmax):每类得分 = 偏置 + 词权重乘计数
    s = [math.fsum([W[k][V]] + [W[k][j] * c for j, c in x])
         for k in range(K)]
    m = max(s)
    e = [math.exp(v - m) for v in s]
    z = math.fsum(e)
    return [v / z for v in e]


def sgd(W, xs, ys, idx, epochs, lr0, rng, lam=0.002):
    # 随机梯度下降:每步只更新一次权重(单步状态更新)
    order = list(idx)
    for ep in range(epochs):
        lr = lr0 / (1 + ep)
        rng.shuffle(order)
        for i in order:
            p = probs(W, xs[i])
            for k in range(K):
                g = p[k] - (1.0 if ys[i] == k else 0.0)
                row = W[k]
                for j, c in xs[i]:
                    row[j] -= lr * (g * c + lam * row[j])
                row[V] -= lr * g
    return W


def fit(xs, ys, idx):
    W = [[0.0] * (V + 1) for _ in range(K)]
    return sgd(W, xs, ys, idx, 10, 0.3, random.Random(7))


def argmax(p):
    return p.index(max(p))


def accuracy(W, xs, ys):
    hit = 0
    for x, y in zip(xs, ys):
        hit += argmax(probs(W, x)) == y
    return hit / len(xs)


pool, test = make_data(1, 6000), make_data(2, 3000)
px, py = pool
rng = random.Random(11)
order = list(range(6000))
rng.shuffle(order)
A = fit(px, py, order[:160])
B = fit(px, py, order[160:320])
rest = order[320:]
pa = [argmax(probs(A, px[i])) for i in rest]
pb = [argmax(probs(B, px[i])) for i in rest]
yy = [py[i] for i in rest]
n = len(rest)
dis = [t for t in range(n) if pa[t] != pb[t]]
only_a = [t for t in dis if pa[t] == yy[t]]
only_b = [t for t in dis if pb[t] == yy[t]]
acc_a = [pa[t] == yy[t] for t in range(n)].count(True) / n
acc_b = [pb[t] == yy[t] for t in range(n)].count(True) / n
d = (len(only_a) + len(only_b)) / n
a = len(only_a) / (len(only_a) + len(only_b))
print("accuracy on {:,} unlabeled: A {:.3f}  B {:.3f}  gap {:.3f}".format(
    n, acc_a, acc_b, acc_a - acc_b))
print("A and B disagree on {:.3f} of messages".format(len(dis) / n))
print("exactly one is right on d = {:.3f} of messages".format(d))
print("  of those, A is right on a = {:.3f}, B on {:.3f}".format(a, 1 - a))
Z = NormalDist()
za, zb = Z.inv_cdf(0.975), Z.inv_cdf(0.80)
m = ((za * 0.5 + zb * math.sqrt(a * (1 - a))) / (a - 0.5)) ** 2
q = len(dis) / n
print("informative labels needed: {}".format(math.ceil(m)))
print("labeling disagreements only: about {} labels".format(
    math.ceil(m * q / d)))
print("labeling random messages: about {:,} labels".format(
    math.ceil(m / d)))
print("share of labels the disagreement filter saves: {:.3f}".format(
    1 - q))

Single source and citation

Fan, Y. (2026). Active Learning and Uncertainty Sampling: the Fewest Human Labels a Customer-Intent Classifier Needs to Reach Its Target (Version 1.0.0), Chapter 7: Eastern and Western Classics: the Xueji's Easy-First Order and Bacon's Crucial Instances. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-312 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026312,
  author    = {Fan, Yafu},
  title     = {Active Learning and Uncertainty Sampling: the Fewest Human Labels a Customer-Intent Classifier Needs to Reach Its Target},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book