What does Three-Light Cut Points conclude, in numbers, about “Big Sales and Drift: Bayesian Correction for Prior Shift and the Sampling Error of Cut Points”?
Answer: Chapter 5 treats the prior shift of a big sale: when the true share of right answers falls to 0.600, the bot's mean self-reported chance is still 0.843.
Scope of the numbers: Private customer service has a third action, a bot draft checked by a person, so it needs a six-cell loss matrix with two cut points; under illustrative parameters red lights below a chance of being right of 0.4000 and green from 0.9500. Parameters are illustrative, all conversations are fictional, and all numbers come from the book's programs, which read and write no file and connect to no customer-service system.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.600, 0.843
| calibration-month base rate pi = 0.897 | ||||||
| pi_new | EM est | mean stale p | stale | adjusted | oracle | green |
| 0.897 | 0.905 | 0.899 | 238.3 | 238.6 | 238.3 | 0.537 |
| 0.850 | 0.843 | 0.886 | 308.6 | 303.2 | 303.2 | 0.362 |
| 0.800 | 0.815 | 0.879 | 372.8 | 352.0 | 351.8 | 0.328 |
| 0.700 | 0.692 | 0.854 | 525.2 | 459.2 | 459.2 | 0.166 |
| 0.600 | 0.632 | 0.843 | 663.0 | 533.6 | 533.0 | 0.122 |
Show the full program
# Chapter 5. p51: a promotion changes the base rate, not the scores.
# Model (illustrative): in the calibration month the bot is right with
# p = 1 / (1 + e^-z), z ~ N(3.0, 1.6), base rate pi = 0.897. In a new
# month the share of chats it can handle drops to pi_new while the
# score distribution within "right" and within "wrong" chats stays the
# same (label shift). The posterior then moves by Bayes' rule:
# p' = r p / (r p + s (1 - p)), r = pi_new / pi, s = (1 - pi_new) /
# (1 - pi). Saerens et al. (2002) estimate pi_new by EM from unlabelled
# scores alone. 3000 unlabelled chats per month (seed 20261113); loss
# matrix of p21; yuan per 100 chats under the true new posterior.
import math
import random
rng = random.Random(20261113)
LW, CA, M, CR = 50.0, 2.0, 0.8, 8.0
P_HI = 1 - CA / (M * LW)
P_LO = 1 - (CR - CA) / ((1 - M) * LW)
def sig(z):
return 1 / (1 + math.exp(-z))
def base_rate(n=4001):
zs = [3.0 + 1.6 * (-8.0 + 16.0 * i / (n - 1)) for i in range(n)]
ws = [math.exp(-0.5 * ((z - 3.0) / 1.6) ** 2) for z in zs]
return math.fsum(w * sig(z) for z, w in zip(zs, ws)) / math.fsum(ws)
PI = base_rate()
def draw_class(y):
while True:
z = rng.gauss(3.0, 1.6)
q = sig(z)
if rng.random() < (q if y == 1 else 1 - q):
return q
def shift(p, pi_new):
r, s = pi_new / PI, (1 - pi_new) / (1 - PI)
return r * p / (r * p + s * (1 - p))
def em(ps):
est = PI
for _ in range(200):
est = math.fsum(shift(p, est) for p in ps) / len(ps)
return est
def risk(q, p):
if q >= P_HI:
return (1 - p) * LW
if q >= P_LO:
return CA + (1 - p) * (1 - M) * LW
return CR
print("calibration-month base rate pi = %.3f" % PI)
print("pi_new EM est mean stale p stale adjusted oracle green")
for pi_new in (0.897, 0.85, 0.8, 0.7, 0.6):
ps = [draw_class(1 if rng.random() < pi_new else 0)
for _ in range(3000)]
est = em(ps)
true_post = [shift(p, pi_new) for p in ps]
cost = []
for f in (lambda p: p, lambda p: shift(p, est),
lambda p: shift(p, pi_new)):
cost.append(100 * math.fsum(risk(f(p), t)
for p, t in zip(ps, true_post))
/ len(ps))
green = len([1 for p in ps if shift(p, est) >= P_HI]) / len(ps)
print("%6.3f %6.3f %12.3f %5.1f %8.1f %6.1f %5.3f"
% (pi_new, est, math.fsum(ps) / len(ps), cost[0], cost[1],
cost[2], green))Single source and citation
Fan, Y. (2026). Three Lights for Handing Over between Bot and Human in Private Customer Service: Misclassification Loss Matrices, Chow's Reject Rule and Optimal Cut Points on Calibrated Confidence (Version 1.0.0), Chapter 5: Big Sales and Drift: Bayesian Correction for Prior Shift and the Sampling Error of Cut Points. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-266 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026266,
author = {Fan, Yafu},
title = {Three Lights for Handing Over between Bot and Human in Private Customer Service: Misclassification Loss Matrices, Chow's Reject Rule and Optimal Cut Points on Calibrated Confidence},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · The Blind Spot of Accuracy: Chow's Reject Rule and the Cost Ratio
- Chapter 2 · A Six-Cell Loss Matrix: Bayes Risk in Three Zones and Two Cut Points
- Chapter 3 · Cost Curves: Why the More Accurate Bot Can Cost More
- Chapter 4 · Confidence Is Not Probability: Isotonic Calibration and the Price of Few Labels
- Chapter 6 · Red Lights Beyond Price: Stop Means Stop, Vulnerable Customers Go to Staff, and the Staffing Cap
- Chapter 7 · Setting the Doubtful Aside, and Bayes: Two Classical Texts on Being Unsure
- Chapter 8 · The Monthly Recalculation Loop, the Distilled Tool and a Falsification Plan