关于「几千条与几百条:不确定性采样的反常识账本」,《主动学习与客服意图标注》给出的定量结论是什么?

结论:第一章在合成的四意图客服数据上重算:达到目标准确率,随机抽样的中位数是 1,600 条,最小间隔采样是 640 条,后者只用前者的 0.400;全部标注训练的模型准确率为 0.893,这一倍数随目标逼近天花板而变化。

数据口径:书中意图先验、词概率与成本均为示意参数;全部计算由书中程序生成;本书属于非个人化经营建议。 第一章在合成的四意图客服数据上重算:达到目标准确率,随机抽样的中位数是 1,600 条,最小间隔采样是 640 条,后者只用前者的 0.400;全部标注训练的模型准确率为 0.893,这一倍数随目标逼近天花板而变化。

English version

对照数据表

下表数字由本章 Python 程序原样打印,可重新运行复现。 程序只用 Python 标准库与结论吻合的打印值: 0.400, 0.893, 1600, 640

Full pool of 6,000 labels: test accuracy 0.893
labelsrandom(mean of 3)margin(mean of 3)
800.7680.810
1600.8150.840
3200.8510.867
6400.8680.882
12800.8800.886
19200.8830.887
25600.8860.889
32000.8830.889
labels to reach 0.880, seeds 11 12 13:
random 1280 1920 1600
margin 800 480 640
median: random 1,600 vs margin 640; margin uses 0.400 of it
查看完整验算程序
# Chapter 1. p12: learning curves, random versus margin sampling.
# 合成客服意图数据(固定种子):池 6,000 条、测试 3,000 条、4 个意图。
# 模型:多类逻辑回归,随机梯度下降,每轮从零重训;参数全部为示意。
# 两种策略各跑 3 个种子,记录首次达到目标准确率 0.880 所需标注数。
import math
import random

WORDS = (
    "where track tracking ship shipped late delay courier parcel arrive "
    "refund money back charge charged return cancel paid reimburse "
    "credit size swap small large color exchange fit tight wrong "
    "replace agent human person manager complain angry supervisor "
    "call speak rude order please hi hello thanks item today help "
    "not my the is it can you want").split()
INTENTS = ("track", "refund", "exchange", "human")
V, K = len(WORDS), 4
PRIOR = (0.62, 0.22, 0.11, 0.05)   # 意图先验(示意)
PS, PC = 0.40, 0.10                # 本意图词、易混意图词的概率(示意)
CONF = (1, 0, 1, 1)                # 每个意图最易混的另一意图
ZW = [1.0 / (r + 1) for r in range(10)]   # 意图词内部的齐普夫权重


def make_doc(rng, k):
    # 一条去标识化客服消息:3 到 8 个词,表示为(词号, 次数)
    cnt = {}
    for _ in range(rng.randint(3, 8)):
        u = rng.random()
        if u < PS:
            j = rng.choices(range(10 * k, 10 * k + 10), ZW)[0]
        elif u < PS + PC:
            c = CONF[k]
            j = rng.choices(range(10 * c, 10 * c + 10), ZW)[0]
        else:
            j = rng.randrange(40, V)
        cnt[j] = cnt.get(j, 0) + 1
    return tuple(sorted(cnt.items()))


def make_data(seed, n):
    rng = random.Random(seed)
    xs, ys = [], []
    for _ in range(n):
        k = rng.choices(range(K), PRIOR)[0]
        xs.append(make_doc(rng, k))
        ys.append(k)
    return xs, ys


def probs(W, x):
    # 多类逻辑回归(softmax):每类得分 = 偏置 + 词权重乘计数
    s = [math.fsum([W[k][V]] + [W[k][j] * c for j, c in x])
         for k in range(K)]
    m = max(s)
    e = [math.exp(v - m) for v in s]
    z = math.fsum(e)
    return [v / z for v in e]


def sgd(W, xs, ys, idx, epochs, lr0, rng, lam=0.002):
    # 随机梯度下降:每步只更新一次权重(单步状态更新)
    order = list(idx)
    for ep in range(epochs):
        lr = lr0 / (1 + ep)
        rng.shuffle(order)
        for i in order:
            p = probs(W, xs[i])
            for k in range(K):
                g = p[k] - (1.0 if ys[i] == k else 0.0)
                row = W[k]
                for j, c in xs[i]:
                    row[j] -= lr * (g * c + lam * row[j])
                row[V] -= lr * g
    return W


def fit(xs, ys, idx):
    W = [[0.0] * (V + 1) for _ in range(K)]
    return sgd(W, xs, ys, idx, 10, 0.3, random.Random(7))


def argmax(p):
    return p.index(max(p))


def accuracy(W, xs, ys):
    hit = 0
    for x, y in zip(xs, ys):
        hit += argmax(probs(W, x)) == y
    return hit / len(xs)


def score(strategy, p):
    # 越小越优先标注
    a = sorted(p, reverse=True)
    if strategy == "margin":
        return a[0] - a[1]
    if strategy == "least":
        return a[0]
    if strategy == "entropy":
        return math.fsum(v * math.log(v) for v in p if v > 0)
    return -a[0]                     # "easy": 最有把握的先标


GRID = (20, 30, 40, 60, 80, 100, 120, 160, 200, 240, 320, 400, 480,
        640, 800, 960, 1280, 1600, 1920, 2560, 3200)


def run(strategy, seed, pool, test, top):
    # 返回 {标注数: 测试准确率} 与已标注下标(按标注先后)
    px, py = pool
    rng = random.Random(seed)
    order = list(range(len(px)))
    rng.shuffle(order)
    lab = order[:GRID[0]]
    taken = set(lab)
    W = fit(px, py, lab)
    curve = {GRID[0]: accuracy(W, *test)}
    for g in GRID[1:]:
        if g > top:
            break
        need = g - len(lab)
        if strategy == "random":
            cand = order[len(lab):g]
        else:
            sc = sorted((score(strategy, probs(W, px[i])), i)
                        for i in range(len(px)) if i not in taken)
            cand = [i for _, i in sc[:need]]
        taken.update(cand)
        lab = lab + cand
        W = fit(px, py, lab)
        curve[g] = accuracy(W, *test)
    return curve, lab, W


def first_reach(curve, target):
    for g in sorted(curve):
        if curve[g] >= target:
            return g
    return None


pool, test = make_data(1, 6000), make_data(2, 3000)
full = accuracy(fit(pool[0], pool[1], range(6000)), *test)
SEEDS = (11, 12, 13)
res = {s: [run(s, sd, pool, test, 3200)[0] for sd in SEEDS]
       for s in ("random", "margin")}
print("Full pool of 6,000 labels: test accuracy {:.3f}".format(full))
print("labels   random(mean of 3)   margin(mean of 3)")
for g in (80, 160, 320, 640, 1280, 1920, 2560, 3200):
    r = math.fsum(c[g] for c in res["random"]) / 3
    m = math.fsum(c[g] for c in res["margin"]) / 3
    print("{:6d}        {:.3f}               {:.3f}".format(g, r, m))
T = 0.880
need = {s: [first_reach(c, T) for c in res[s]] for s in res}
print("labels to reach {:.3f}, seeds 11 12 13:".format(T))
for s in ("random", "margin"):
    print("  {:<6s} {}".format(s, " ".join(str(g) for g in need[s])))
mr = sorted(need["random"])[1]
mm = sorted(need["margin"])[1]
print("median: random {:,} vs margin {}; margin uses {:.3f} of it"
      .format(mr, mm, mm / mr))

唯一数据源与引用

范雅福(Fan, Y.). (2026). 《主动学习与不确定性采样、最少人工标注让客服意图分类器达标母典》第 1 章「几千条与几百条:不确定性采样的反常识账本」(版本 1.0.0). 5D Library. DOI 待发布

DOI 待发布 · 专著页正式发布后上线(本站暂无专著页) · 5DL-B-312 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026312,
  author    = {Fan, Yafu},
  title     = {Active Learning and Uncertainty Sampling: the Fewest Human Labels a Customer-Intent Classifier Needs to Reach Its Target},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

本书其他章节结论