What does Three-Light Cut Points conclude, in numbers, about “Cost Curves: Why the More Accurate Bot Can Cost More”?

Answer: Chapter 3 compares two bots on cost curves; the less accurate bot that rejects better is cheaper above a cost ratio of 4.14.

Scope of the numbers: Private customer service has a third action, a bot draft checked by a person, so it needs a six-cell loss matrix with two cut points; under illustrative parameters red lights below a chance of being right of 0.4000 and green from 0.9500. Parameters are illustrative, all conversations are fictional, and all numbers come from the book's programs, which read and write no file and connect to no customer-service system.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 4.14

bot A accuracy 0.897AUC 0.842
bot B accuracy 0.819AUC 0.957
LWLW/CRcost Acost Bcheaper
101.25102.8169.1A
162.00158.0211.0A
243.00217.1244.4A
324.00264.4267.6A
506.25343.7303.3B
8010.00430.5340.9B
12816.00514.8378.5B
25632.00623.4433.4B
break-even wrong-answer loss LW = 33.1 (LW/CR = 4.14)
Show the full program
# Chapter 3. p31: the more accurate bot is not always the cheaper one.
# Model (illustrative): two calibrated bots, p = 1 / (1 + e^-z).
# Bot A: z ~ N(3.0, 1.6), the p11 bot. Bot B: z ~ N(4.0, 4.0), less
# accurate overall but with more of its mass at the extremes. Each
# uses its own Chow cut 1 - CR / LW with CR = 8; the table gives
# yuan per 100 chats as LW moves, i.e. a cost curve in the spirit of
# Drummond and Holte (2006). AUC is the chance that a chat the bot
# gets right outranks one it gets wrong (exact, by quadrature).
import math

CR = 8.0


def nodes(mu, sd, n=4001):
    zs = [mu + sd * (-8.0 + 16.0 * i / (n - 1)) for i in range(n)]
    ws = [math.exp(-0.5 * ((z - mu) / sd) ** 2) for z in zs]
    tot = math.fsum(ws)
    return [(1.0 / (1.0 + math.exp(-z)), w / tot) for z, w in zip(zs, ws)]


def auc(pw):
    pr = math.fsum(w * p for p, w in pw)
    below, terms = [], []
    for p, w in pw:  # nodes are sorted by p already
        terms.append(w * p * math.fsum(below))
        below.append(w * (1 - p))
    return math.fsum(terms) / (pr * (1 - pr))


def cost(pw, lw):
    return 100 * math.fsum(w * min((1 - p) * lw, CR) for p, w in pw)


A, B = nodes(3.0, 1.6), nodes(4.0, 4.0)
print("bot A accuracy %.3f  AUC %.3f" % (
    math.fsum(w * p for p, w in A), auc(A)))
print("bot B accuracy %.3f  AUC %.3f" % (
    math.fsum(w * p for p, w in B), auc(B)))
print("    LW  LW/CR   cost A   cost B  cheaper")
for lw in (10.0, 16.0, 24.0, 32.0, 50.0, 80.0, 128.0, 256.0):
    ca, cb = cost(A, lw), cost(B, lw)
    print("%6.0f  %5.2f  %7.1f  %7.1f  %7s"
          % (lw, lw / CR, ca, cb, "A" if ca <= cb else "B"))
lo, hi = 10.0, 128.0
for _ in range(60):
    mid = (lo + hi) / 2
    if cost(A, mid) <= cost(B, mid):
        lo = mid
    else:
        hi = mid
print("break-even wrong-answer loss LW = %.1f (LW/CR = %.2f)"
      % (lo, lo / CR))

Single source and citation

Fan, Y. (2026). Three Lights for Handing Over between Bot and Human in Private Customer Service: Misclassification Loss Matrices, Chow's Reject Rule and Optimal Cut Points on Calibrated Confidence (Version 1.0.0), Chapter 3: Cost Curves: Why the More Accurate Bot Can Cost More. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-266 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026266,
  author    = {Fan, Yafu},
  title     = {Three Lights for Handing Over between Bot and Human in Private Customer Service: Misclassification Loss Matrices, Chow's Reject Rule and Optimal Cut Points on Calibrated Confidence},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book