What does Three-Light Cut Points conclude, in numbers, about “Confidence Is Not Probability: Isotonic Calibration and the Price of Few Labels”?
Answer: The optimal loss under the true probability is 240.0 yuan per hundred conversations. With only 50 labels, isotonic regression loses an extra 38.4 yuan on average and Platt scaling 12.9 yuan; with 100 labels, 22.5 and 6.6 yuan; with 300, 7.6 and 1.9; with 1,000, 3.6 and 0.8; with 3,000, 1.8 and 0.3. The mean gap from the true probability tells the same story: 0.0687 for isotonic and 0.0382 for Platt with 50 labels, falling to 0.0145 and 0.0051 with 3,000.
Scope of the numbers: Private customer service has a third action, a bot draft checked by a person, so it needs a six-cell loss matrix with two cut points; under illustrative parameters red lights below a chance of being right of 0.4000 and green from 0.9500. Parameters are illustrative, all conversations are fictional, and all numbers come from the book's programs, which read and write no file and connect to no customer-service system.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0051, 0.0145, 0.0382, 0.0687, 0.3, 0.8
| oracle cost per 100 chats: 240.0 | ||||
| n | iso regret | Platt regret | iso |q-p| | Platt |q-p| |
| 50 | 38.4 | 12.9 | 0.0687 | 0.0382 |
| 100 | 22.5 | 6.6 | 0.0560 | 0.0297 |
| 300 | 7.6 | 1.9 | 0.0336 | 0.0151 |
| 1,000 | 3.6 | 0.8 | 0.0218 | 0.0097 |
| 3,000 | 1.8 | 0.3 | 0.0145 | 0.0051 |
Show the full program
# Chapter 4. p43: with few labels, isotonic overfits; Platt holds up.
# Model (illustrative): the p41 world and the p21 loss matrix. For each
# training size n, 40 repeats (seed 20261113) fit isotonic and Platt
# maps; each is scored on one fixed set of 4000 chats by the expected
# loss under the true p (yuan per 100 chats, regret over the oracle)
# and by the mean absolute gap |q - p| between map and truth. Platt
# uses smoothed targets (n+ + 1) / (n+ + 2) and 1 / (n- + 2). The
# Platt family here contains the truth (b = 1 / 1.6), which favours it.
import bisect
import math
import random
rng = random.Random(20261113)
LW, CA, M, CR = 50.0, 2.0, 0.8, 8.0
P_HI = 1 - CA / (M * LW)
P_LO = 1 - (CR - CA) / ((1 - M) * LW)
def draw(n):
out = []
for _ in range(n):
z = rng.gauss(3.0, 1.6)
p = 1 / (1 + math.exp(-z))
y = 1 if rng.random() < p else 0
out.append((1 / (1 + math.exp(-1.6 * z)), y, p))
return out
def pav(data):
blocks = []
for s, y in sorted(data):
blocks.append([s, float(y), 1])
while (len(blocks) > 1 and blocks[-2][1] / blocks[-2][2]
>= blocks[-1][1] / blocks[-1][2]):
s2, t2, n2 = blocks.pop()
blocks[-1][1] = math.fsum([blocks[-1][1], t2])
blocks[-1][2] += n2
starts, vals = [b[0] for b in blocks], [b[1] / b[2] for b in blocks]
return lambda s: vals[max(0, bisect.bisect_right(starts, s) - 1)]
def logit(s):
s = min(max(s, 1e-9), 1 - 1e-9)
return math.log(s / (1 - s))
def sig(u):
if u >= 0:
return 1 / (1 + math.exp(-u))
e = math.exp(u)
return e / (1 + e)
def platt(data):
# smoothed targets as in Platt's recipe keep the fit finite
npos = len([1 for _, y in data if y == 1])
tp, tn = (npos + 1) / (npos + 2), 1 / (len(data) - npos + 2)
a, b = 0.0, 1.0
xs = [(logit(s), tp if y == 1 else tn) for s, y in data]
def nll(a, b):
return -math.fsum(t * math.log(sig(a + b * x) + 1e-300)
+ (1 - t) * math.log(sig(-a - b * x) + 1e-300)
for x, t in xs)
for _ in range(25):
g0, g1, h00, h01, h11 = [], [], [], [], []
for x, y in xs:
q = sig(a + b * x)
v = q * (1 - q) + 1e-12
g0.append(y - q)
g1.append((y - q) * x)
h00.append(v)
h01.append(v * x)
h11.append(v * x * x)
G0, G1 = math.fsum(g0), math.fsum(g1)
H00, H01, H11 = math.fsum(h00), math.fsum(h01), math.fsum(h11)
det = H00 * H11 - H01 * H01
da = (H11 * G0 - H01 * G1) / det
db = (H00 * G1 - H01 * G0) / det
step, f0 = 1.0, nll(a, b)
while step > 1e-6 and nll(a + step * da, b + step * db) > f0:
step /= 2 # damped Newton: halve until the fit improves
a, b = a + step * da, b + step * db
return lambda s: sig(a + b * logit(s))
def risk(q, p):
if q >= P_HI:
return (1 - p) * LW
if q >= P_LO:
return CA + (1 - p) * (1 - M) * LW
return CR
test = draw(4000)
oracle = 100 * math.fsum(risk(p, p) for _, _, p in test) / len(test)
print("oracle cost per 100 chats: %.1f" % oracle)
print(" n iso regret Platt regret iso |q-p| Platt |q-p|")
for n in (50, 100, 300, 1000, 3000):
ri, rp, gi, gp = [], [], [], []
for _ in range(40):
tr = [(s, y) for s, y, _ in draw(n)]
fi, fp = pav(tr), platt(tr)
qi = [fi(s) for s, _, _ in test]
qp = [fp(s) for s, _, _ in test]
ri.append(100 * math.fsum(risk(q, r[2]) for q, r in zip(qi, test))
/ len(test) - oracle)
rp.append(100 * math.fsum(risk(q, r[2]) for q, r in zip(qp, test))
/ len(test) - oracle)
gi.append(math.fsum(abs(q - r[2]) for q, r in zip(qi, test))
/ len(test))
gp.append(math.fsum(abs(q - r[2]) for q, r in zip(qp, test))
/ len(test))
print("%6s %10.1f %12.1f %9.4f %11.4f" % (
format(n, ","), math.fsum(ri) / 40, math.fsum(rp) / 40,
math.fsum(gi) / 40, math.fsum(gp) / 40))Single source and citation
Fan, Y. (2026). Three Lights for Handing Over between Bot and Human in Private Customer Service: Misclassification Loss Matrices, Chow's Reject Rule and Optimal Cut Points on Calibrated Confidence (Version 1.0.0), Chapter 4: Confidence Is Not Probability: Isotonic Calibration and the Price of Few Labels. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-266 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026266,
author = {Fan, Yafu},
title = {Three Lights for Handing Over between Bot and Human in Private Customer Service: Misclassification Loss Matrices, Chow's Reject Rule and Optimal Cut Points on Calibrated Confidence},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · The Blind Spot of Accuracy: Chow's Reject Rule and the Cost Ratio
- Chapter 2 · A Six-Cell Loss Matrix: Bayes Risk in Three Zones and Two Cut Points
- Chapter 3 · Cost Curves: Why the More Accurate Bot Can Cost More
- Chapter 5 · Big Sales and Drift: Bayesian Correction for Prior Shift and the Sampling Error of Cut Points
- Chapter 6 · Red Lights Beyond Price: Stop Means Stop, Vulnerable Customers Go to Staff, and the Staffing Cap
- Chapter 7 · Setting the Doubtful Aside, and Bayes: Two Classical Texts on Being Unsure
- Chapter 8 · The Monthly Recalculation Loop, the Distilled Tool and a Falsification Plan