What does Context-Parametric Rule Distillation and Online Calibration conclude, in numbers, about “Calibration and the Whole Pipeline: the Brier Partition, Binned Calibration Error and a Net-Benefit Simulation”?
Answer: Chapter 6 checks calibration with the Brier partition and binned calibration error, re-scaling cuts the error from 0.0210 to 0.0035, and a whole-pipeline simulation shows online calibration without a minimum-sample guard losing to the static pack in 164 of 200 worlds.
Scope of the numbers: Parameters are illustrative and all calculations are generated by the book's programs. This is general business analysis, not investment, legal or medical advice, and it describes no real customer or sale.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0035, 0.0210
| Platt fit on the first half: a | 0.617, b = -1.335 |
| ideal inverse map: a | 0.588, b = -1.412 |
| second half: ECE raw 0.0210, re-scaled 0.0035 | |
| log loss raw 0.1709, re-scaled 0.1643 | |
| bin raw mean f observed | scaled mean f observed | |
| 1 | 0.0072 0.0155 | 0.0123 0.0155 |
| 3 | 0.0197 0.0260 | 0.0230 0.0260 |
| 6 | 0.0440 0.0355 | 0.0379 0.0355 |
| 8 | 0.0754 0.0505 | 0.0530 0.0505 |
| 10 | 0.1993 0.0930 | 0.1001 0.0930 |
Show the full program
# Chapter 6. p62: binned calibration error and Platt re-scaling.
# True pay probability p = s(z), z ~ Normal(-3.3, 0.6); the rule pack
# reports f = s(1.7 z + 2.4), too spread out (illustrative). 20,000
# sessions fit a, b in s(a logit f + b) by Newton steps; 20,000 more
# measure the expected calibration error over 10 equal-count bins
# (ECE = sum over bins of share * |mean f - observed rate|).
import math
import random
def s(x):
return 1.0 / (1.0 + math.exp(-x))
def logit(p):
return math.log(p / (1.0 - p))
def simulate(rng, n):
rows = []
for _ in range(n):
z = rng.gauss(-3.3, 0.6)
y = 1 if rng.random() < s(z) else 0
rows.append((s(1.7 * z + 2.4), y))
return rows
def fit(rows):
a, b = 1.0, 0.0
for _ in range(25):
g = [0.0, 0.0]
h = [0.0, 0.0, 0.0]
ga, gb, haa, hab, hbb = [], [], [], [], []
for f, y in rows:
x = logit(f)
p = s(a * x + b)
ga.append((p - y) * x)
gb.append(p - y)
w = p * (1 - p)
haa.append(w * x * x)
hab.append(w * x)
hbb.append(w)
g = [math.fsum(ga), math.fsum(gb)]
h = [math.fsum(haa), math.fsum(hab), math.fsum(hbb)]
det = h[0] * h[2] - h[1] * h[1]
a -= (h[2] * g[0] - h[1] * g[1]) / det
b -= (h[0] * g[1] - h[1] * g[0]) / det
return a, b
def ece(rows, bins=10):
rows = sorted(rows)
n = len(rows)
parts, table = [], []
for i in range(bins):
chunk = rows[i * n // bins:(i + 1) * n // bins]
mf = math.fsum(f for f, _ in chunk) / len(chunk)
mo = math.fsum(y for _, y in chunk) / len(chunk)
parts.append(len(chunk) / n * abs(mf - mo))
table.append((mf, mo))
return math.fsum(parts), table
def logloss(rows):
return -math.fsum(y * math.log(f) + (1 - y) * math.log(1 - f)
for f, y in rows) / len(rows)
def main():
rng = random.Random(6201)
train, test = simulate(rng, 20000), simulate(rng, 20000)
a, b = fit(train)
print("Platt fit on the first half: a = {:.3f}, b = {:.3f}".format(
a, b))
print("ideal inverse map: a = {:.3f}, b = {:.3f}".format(
1 / 1.7, -2.4 / 1.7))
fixed = [(s(a * logit(f) + b), y) for f, y in test]
e0, t0 = ece(test)
e1, t1 = ece(fixed)
print("second half: ECE raw {:.4f}, re-scaled {:.4f}".format(e0, e1))
print("log loss raw {:.4f}, re-scaled {:.4f}".format(
logloss(test), logloss(fixed)))
print(" bin raw mean f observed | scaled mean f observed")
for i in (0, 2, 5, 7, 9):
print(" {:>2} {:.4f} {:.4f} | {:.4f} {:.4f}".format(
i + 1, t0[i][0], t0[i][1], t1[i][0], t1[i][1]))
if __name__ == "__main__":
main()Single source and citation
Fan, Y. (2026). Next-Generation Knowledge Distillation, Context-Parametric Compilation and Live-Conversion Feedback Calibration (Version 1.0.0), Chapter 6: Calibration and the Whole Pipeline: the Brier Partition, Binned Calibration Error and a Net-Benefit Simulation. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-377 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026377,
author = {Fan, Yafu},
title = {Next-Generation Knowledge Distillation, Context-Parametric Compilation and Live-Conversion Feedback Calibration},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Simpson's Reversal: Why a Rule Without Its Context Can Point the Wrong Way
- Chapter 2 · Context-Parametric Compilation: Squeezing a Decision Table into Decision Lists and Ordered Decision Diagrams
- Chapter 3 · Resolving Conflicts Across Volumes: Do More Rules Make a Pack More Accurate?
- Chapter 4 · Online Bayesian Calibration: Giving Every Rule a Beta Distribution That Grows
- Chapter 5 · Threshold Drift: Discounted Beta Updates, Sliding Windows and a Cumulative-Sum Alarm
- Chapter 7 · Classical Sources: Wheelwright Bian and Aristotle on Equity
- Chapter 8 · Closing the Loop: Field Tools and Checker Specifications, the Risk Ledger and a Pre-registered Test