5D Library › Conversational Closing Automata, Objection-Handling Information Gain and High-Ticket Sales Kinetics · 中文
Chapter 8. A Ledger of Ten Thousand Conversations, Probability Calibration and a Pre-Registered Experiment
A ledger of ten thousand conversations
The model of Chapter 6 is simulated buyer by buyer over 10,000 conversations, with only 10% of buyers ready at the start (most enquiries that open a chat are cold). Four approaches are compared: the best AI-only rule; the best rule with human hand-over; “ask for payment at the first buying signal”; and “hand everyone to a human at once”. Refunds take 8% of revenue, each AI conversation costs RM 3 and each human hand-over RM 250.
Calculation
# Chapter 8. p14: A ledger for 10,000 AI sales conversations (RM 9,800
# offer), simulated buyer by buyer with the model of p11; 10% of buyers
# are ready at the start (most inbound chats are cold). Playbooks: the
# optimal AI-only rule, the optimal rule with human hand-over (RM 250 a
# hand-over), "ask at the first buying signal", and "hand everyone to a
# human at once". Refunds take 8%
# of sales revenue; each AI conversation costs RM 3. Seed 84.
import importlib.util
import random
def load(name):
spec = importlib.util.spec_from_file_location(name, name + ".py")
m = importlib.util.module_from_spec(spec)
spec.loader.exec_module(m)
return m
p11 = load("p11_stopping")
V = p11.V
def play(rng, rule, n=10000, p0=0.1):
sold_ai = sold_h = hand = 0
for _ in range(n):
ready, p, sig = rng.random() < p0, p0, False
for turn in range(60):
a = rule(p, sig, turn)
if a == "ask for payment":
if ready and rng.random() < p11.ASK:
sold_ai += 1
break
if not ready and rng.random() < p11.LEAVE:
break
r, s = p * (1 - p11.ASK), (1 - p) * (1 - p11.LEAVE)
p = r / (r + s)
continue
if a == "hand to a human":
hand += 1
sold_h += rng.random() < p11.HUMAN[0 if ready else 1]
break
if rng.random() < p11.DRIFT:
break
if not ready and rng.random() < p11.NURT:
ready = True
q = p + (1 - p) * p11.NURT
y, n_ = p11.SIG
sig = rng.random() < (y if ready else n_)
p = q * y / (q * y + (1 - q) * n_) if sig else \
q * (1 - y) / (q * (1 - y) + (1 - q) * (1 - n_))
rev = (sold_ai + sold_h) * V
net = rev * 0.92 - hand * 250 - n * 3
return sold_ai, sold_h, hand, net
def policy(cost):
_, act = p11.solve(cost)
return lambda p, sig, t: act[min(p11.N, round(p * p11.N))]
if __name__ == "__main__":
rules = {"AI only (optimal)": policy(1e9),
"AI + human (optimal)": policy(250.0),
"ask at first signal": lambda p, s, t: "ask for payment"
if s else "keep talking",
"everyone to a human": lambda p, s, t: "hand to a human"}
print(" playbook AI sales human sales hand-overs"
" net RM")
for name, rule in rules.items():
a, h, k, net = play(random.Random(84), rule)
print(" {:<22} {:8,d} {:11,d} {:10,d} {:>12,.0f}".format(
name, a, h, k, net))
# playbook AI sales human sales hand-overs net RM
# AI only (optimal) 2,358 0 0 21,229,728
# AI + human (optimal) 0 2,618 6,302 21,998,388
# ask at first signal 1,930 0 0 17,370,880
# everyone to a human 0 2,674 10,000 21,578,784The best rule with human hand-over nets RM 21,998,388, with 6,302 hand-overs and 2,618 sales closed by humans; handing everyone to a human at once nets RM 21,578,784 with 10,000 hand-overs; the best AI-only rule closes 2,358 sales and nets RM 21,229,728; “ask at the first signal” closes only 1,930 and nets RM 17,370,880, the worst of the four. In cold-start conversations the buyer’s first buying signal pushes the belief into the hand-over zone, so under the best rule the AI’s work is qualifying and nurturing, and the human closes.
Is the AI’s “probability of buying” a probability?
The threshold rule of Chapter 6 takes the AI’s probability of a sale as its input. If that number is not a calibrated probability, the thresholds fail. Below, an overconfident score is simulated: the more extreme the true probability, the more extreme the score.
Calculation
# Chapter 8. p15: Is the AI closer's "probability of buying" a probability?
# 8,000 simulated conversations; the true chance t is Beta(2, 8); the
# model reports an overconfident score sigmoid(1.6 logit(t) + 0.3). The
# first 4,000 conversations fit a recalibration (mean outcome per score
# bin of width 0.1); the last 4,000 test it. Brier score = mean squared
# error. Also: how many test conversations cross the ask-for-payment
# threshold of p11 (0.5) on each score, and how many truly should. Seed 84.
import random
from math import exp, log
def sig(x):
return 1 / (1 + exp(-x))
def logit(p):
return log(p / (1 - p))
if __name__ == "__main__":
rng = random.Random(84)
rows = []
for _ in range(8000):
t = rng.betavariate(2, 8)
s = sig(1.6 * logit(t) + 0.3)
rows.append((t, s, 1 if rng.random() < t else 0))
fit, test = rows[:4000], rows[4000:]
bins = {}
for t, s, y in fit:
b = min(int(s * 10), 9)
n, k = bins.get(b, (0, 0))
bins[b] = (n + 1, k + y)
cal = {b: k / n for b, (n, k) in bins.items()}
def recal(s):
b = min(int(s * 10), 9)
return cal.get(b, s)
brier = lambda f: sum((f(t, s) - y) ** 2 for t, s, y in test) / 4000
print(" Brier score: raw {:.4f}, recalibrated {:.4f}, truth {:.4f}"
.format(brier(lambda t, s: s), brier(lambda t, s: recal(s)),
brier(lambda t, s: t)))
print(" score bin conversations mean score share who bought")
for b in range(0, 10, 2):
sel = [(s, y) for t, s, y in test if b / 10 <= s < (b + 2) / 10]
if sel:
print(" {:.1f}-{:.1f} {:13d} {:10.3f} {:16.3f}".format(
b / 10, (b + 2) / 10, len(sel),
sum(s for s, _ in sel) / len(sel),
sum(y for _, y in sel) / len(sel)))
over = [sum(1 for t, s, y in test if f(t, s) >= 0.5)
for f in (lambda t, s: s, lambda t, s: recal(s),
lambda t, s: t)]
print(" above the 0.5 threshold: raw {}, recalibrated {}, truly {}"
.format(*over))
# Brier score: raw 0.1452, recalibrated 0.1439, truth 0.1422
# score bin conversations mean score share who bought
# 0.0-0.2 2874 0.079 0.132
# 0.2-0.4 810 0.280 0.309
# 0.4-0.6 248 0.479 0.419
# 0.6-0.8 64 0.665 0.609
# 0.8-1.0 4 0.822 0.500
# above the 0.5 threshold: raw 157, recalibrated 68, truly 83On 4,000 test conversations the raw score has a Brier score of 0.1452; recalibrated by score bins it has 0.1439; the true probability itself has 0.1422 (Brier 1950). The gaps look small, but the threshold decisions differ greatly: 157 conversations cross the ask-for-payment threshold of 0.5 on the raw score, 68 after recalibration, and 83 truly should. The raw score would have the AI ask for money too early in about twice as many conversations.
Note
A four-step check. First, ask diagnostic questions until no answer can change the decision, not in order of bits. Second, decode the cause of each of the four objections, ask a clarifying question after “too expensive”, and ban fake deadlines and repeated pushing (Parliament of Malaysia 1999). Third, rank conversations for hand-over by expected sale value with a human or by uplift, not by warmth; give new sources more human time by their Gittins index. Fourth, recalibrate the AI’s sale probabilities against actual outcomes every month, with refund and chargeback rates as guardrails (Kohavi et al. 2020).
A pre-registered experiment
Calculation
# Chapter 8. p16: Sample sizes for a pre-registered test (two-sided alpha
# 0.05, power 0.80). (1) Close rate now: pitch at once 0.108 against one
# diagnostic question first 0.138 (p02). (2) Value per lead: RM 2,522
# against RM 2,748 (p03); a lead's value is lumpy (most buy nothing), sd
# about RM 7,000. (3) Net per AI conversation: AI only RM 2,123 against
# AI with human hand-over RM 2,200 (p14), sd about RM 4,000.
from math import ceil, sqrt
ZA, ZB = 1.96, 0.84
def two_props(p1, p2):
pb = (p1 + p2) / 2
num = (ZA * sqrt(2 * pb * (1 - pb)) +
ZB * sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
return ceil(num / (p1 - p2) ** 2)
def two_means(m1, m2, sd):
return ceil(2 * ((ZA + ZB) * sd / (m2 - m1)) ** 2)
if __name__ == "__main__":
print(" (1) close rate 0.108 vs 0.138: {:,} leads per arm".format(
two_props(0.108, 0.138)))
print(" (2) value RM 2,522 vs 2,748: {:,} leads per arm".format(
two_means(2522, 2748, 7000)))
print(" (3) net RM 2,123 vs 2,200: {:,} conversations per arm".format(
two_means(2123, 2200, 4000)))
# (1) close rate 0.108 vs 0.138: 1,879 leads per arm
# (2) value RM 2,522 vs 2,748: 15,043 leads per arm
# (3) net RM 2,123 vs 2,200: 42,315 conversations per armTesting the effect of “one diagnostic question first” on the close rate now (0.108 against 0.138) needs 1,879 enquirers per arm; testing the value per enquirer (RM 2,522 against RM 2,748), because value is very uneven, needs 15,043 per arm; testing the difference in net revenue between hand-over rules (RM 2,123 against RM 2,200 per conversation) needs 42,315 conversations per arm.
Hypothesis · not established
Pre-registered prediction (not yet tested): asking one situation question before quoting raises the close rate now by at least 2 percentage points over quoting at once; handing over by a belief threshold nets no less per conversation than handing everyone to a human at once. What would show it wrong: the former’s close-rate gap is significantly below 2 points, or the latter’s net revenue is significantly lower.
Summary of the book
A high-ticket private-channel conversation is a partially observable decision problem. An early pitch is not empty of information; its cost is the buyers lost after a refusal. The value of diagnostic questions lies in changing decisions: the optimal question tree asks fewer than two questions on average, and the full-information bound is 37% above pitching at once. Each of the four objections has different causes, and answering by the posterior recovers more than a third more than one script; high pressure nets out negative once refunds, chargebacks and reputation are counted. The division of work between AI and humans is set by a trust-threshold stopping rule, hand-overs should go to the buyers a human can add most to, the human desk should be staffed as a queue, and the AI’s probabilities must be calibrated.
References
Fan, Y. (2026). Conversational Closing Automata, Objection-Handling Information Gain and High-Ticket Sales Kinetics: Diagnostic Question Trees, Trust-Threshold Stopping Rules and AI Closer Escalation Protocols [对话式成交自动机、异议消解信息增益与高客单销售动力学] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090221 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).