What does Active Learning for Intent Labels conclude, in numbers, about “Labeling Budgets, Stopping Rules and Audit Sample Sizes”?

Answer: If the aim is to show that accuracy is at least 0.850 when the true value is 0.880, a one-sided test at significance 0.05 with power 0.80 needs 824 messages. In terms of the Wilson lower bound, with an observed 0.880 the bound reaches 0.850 at 550 messages, 0.860 at 1,160 and 0.870 at 4,350. The last line of the program corresponds to S2 above: an audit of 400 with an observed 0.880 has a lower bound of only 0.844, below 0.850, so the rule does not fire.

Scope of the numbers: Intent priors, word probabilities and costs are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (23 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.05, 0.844, 0.850, 0.860, 0.870, 0.880

half-width h audit n (p0.880, 95 %)
0.05163
0.03451
0.021015
0.014057
one-sided test of 0.850 vs true 0.880: n824
Wilson low bound >0.850 at observed 0.880: n = 550
Wilson low bound >0.860 at observed 0.880: n = 1,160
Wilson low bound >0.870 at observed 0.880: n = 4,350
audit of 400 at observed 0.880: low bound 0.844
Show the full program
# Chapter 6. p63: how many random audit labels a claim of accuracy needs.
# 一:估计准确率 p 的 95 % 区间半宽 h:n = z^2 p (1 - p) / h^2。
# 二:单侧检验「准确率不低于 0.850」,真值 0.880,alpha 0.05、功效 0.80。
# 三:Wilson 下界不低于 0.850 所需 n(点估计恰为 0.880 时),逐个搜索。
# 审计样本必须从客流中随机抽取,不能取主动学习挑出的消息。
import math
from statistics import NormalDist

Z = NormalDist()
z975, z95, z80 = Z.inv_cdf(0.975), Z.inv_cdf(0.95), Z.inv_cdf(0.80)
p = 0.880
print("half-width h   audit n (p = 0.880, 95 %)")
for h in (0.05, 0.03, 0.02, 0.01):
    print("   {:.2f}         {:5d}".format(
        h, math.ceil(z975 ** 2 * p * (1 - p) / h ** 2)))
p0, p1 = 0.850, 0.880
n = ((z95 * math.sqrt(p0 * (1 - p0)) + z80 * math.sqrt(p1 * (1 - p1)))
     / (p1 - p0)) ** 2
print("one-sided test of 0.850 vs true 0.880: n = {}".format(math.ceil(n)))


def wilson_low(ph, n, z=z975):
    c = ph + z * z / (2 * n)
    r = z * math.sqrt(ph * (1 - ph) / n + z * z / (4 * n * n))
    return (c - r) / (1 + z * z / n)


for floor in (0.850, 0.860, 0.870):
    m = 10
    while wilson_low(p, m) < floor:
        m += 10
    print("Wilson low bound >= {:.3f} at observed 0.880: n = {:,}".format(
        floor, m))
print("audit of 400 at observed 0.880: low bound {:.3f}".format(
    wilson_low(p, 400)))

Single source and citation

Fan, Y. (2026). Active Learning and Uncertainty Sampling: the Fewest Human Labels a Customer-Intent Classifier Needs to Reach Its Target (Version 1.0.0), Chapter 6: Labeling Budgets, Stopping Rules and Audit Sample Sizes. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-312 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026312,
  author    = {Fan, Yafu},
  title     = {Active Learning and Uncertainty Sampling: the Fewest Human Labels a Customer-Intent Classifier Needs to Reach Its Target},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book