Active Learning and Uncertainty Sampling: the Fewest Human Labels a Customer-Intent Classifier Needs to Reach Its Target: chapter answers
Online shop owners commonly believe that training a customer-intent classifier takes a few thousand randomly drawn past messages to label, and that more is always better. The counterintuitive claim tested here is that the accuracy random labeling reaches only after more than a thousand labels can often be reached with a fraction of them if the model picks the messages it is least sure of and hands those to a person; but messages picked this way carry a sampling bias, and using them to estimate intent shares, to grade the model or to decide when to stop goes wrong. After checking the sources t…
DOI pending · monograph page goes live after publication · 5DL-B-312 · 中文版
- Chapter 1 · Thousands versus Hundreds: the Counterintuitive Ledger of Uncertainty Sampling
Chapter 1 recomputes the comparison on synthetic customer data with four intents: to reach the target accuracy random sampling needs a median of 1,600 labels and margin sampling 640, a ratio of 0.400; the model trained… - Chapter 2 · A Hand-Written Multinomial Logistic Regression: Stochastic Gradient Descent on Integer Bag-of-Words Features
A hand-written logistic regression together with synthetic data whose ceiling can be computed exactly gives a clean test bench: the gradient has passed a numerical check, the training path is repeatable, the ceiling is… - Chapter 3 · Three Rulers of Doubt: Least Confidence, Smallest Margin and Entropy
On the book's data the first hazard is mild: of the 200 messages one margin batch picks, 197 are distinct, and the whole pool of 6,000 messages contains 133 exact duplicates. Yet the make-up of the batch is quite unlike… - Chapter 4 · Sampling Bias: Picked Messages No Longer Stand for the Traffic
Chapter 4 measures the bias: the margin-picked labeled set lies at a total variation distance of 0.208 from the traffic, and a model graded on the messages it picked scores only 0.463 against a true accuracy of 0.876; i… - Chapter 5 · Importance Weighting and Prior Adjustment: Computing the Bias Away
Chapter 5 draws with known inclusion probabilities and weights by their inverse, after Horvitz and Thompson, bringing the grade of a model whose true accuracy is 0.809 back from the plain mean of 0.657 to the truth, and… - Chapter 6 · Labeling Budgets, Stopping Rules and Audit Sample Sizes
If the aim is to show that accuracy is at least 0.850 when the true value is 0.880, a one-sided test at significance 0.05 with power 0.80 needs 824 messages. In terms of the Wilson lower bound, with an observed 0.880 th… - Chapter 7 · Eastern and Western Classics: the Xueji's Easy-First Order and Bacon's Crucial Instances
Program p72 trains two candidate models, A and B, each on 160 random labels that do not overlap, and asks how many labels are needed to decide which is more accurate. The two models disagree on only 0.201 of the message… - Chapter 8 · Closing the Loop: a Labeling Queue for Customer Intents and a Plan to Refute the Claim
Chapter 8 gives a labeling queue for customer intents, the distilled Labeling Priority Sorter and a pre-registered test; the classifier only routes and suggests a hand-over and takes no adverse decision about a customer…
Illustrative model calculation, not a live quote