What does Context-Parametric Rule Distillation and Online Calibration conclude, in numbers, about “Calibration and the Whole Pipeline: the Brier Partition, Binned Calibration Error and a Net-Benefit Simulation”?

Answer: Chapter 6 checks calibration with the Brier partition and binned calibration error, re-scaling cuts the error from 0.0210 to 0.0035, and a whole-pipeline simulation shows online calibration without a minimum-sample guard losing to the static pack in 164 of 200 worlds.

Scope of the numbers: Parameters are illustrative and all calculations are generated by the book's programs. This is general business analysis, not investment, legal or medical advice, and it describes no real customer or sale.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0035, 0.0210

Platt fit on the first half: a0.617, b = -1.335
ideal inverse map: a0.588, b = -1.412
second half: ECE raw 0.0210, re-scaled 0.0035
log loss raw 0.1709, re-scaled 0.1643
bin raw mean f observed | scaled mean f observed
10.0072 0.0155 | 0.0123 0.0155
30.0197 0.0260 | 0.0230 0.0260
60.0440 0.0355 | 0.0379 0.0355
80.0754 0.0505 | 0.0530 0.0505
100.1993 0.0930 | 0.1001 0.0930
Show the full program
# Chapter 6. p62: binned calibration error and Platt re-scaling.
# True pay probability p = s(z), z ~ Normal(-3.3, 0.6); the rule pack
# reports f = s(1.7 z + 2.4), too spread out (illustrative). 20,000
# sessions fit a, b in s(a logit f + b) by Newton steps; 20,000 more
# measure the expected calibration error over 10 equal-count bins
# (ECE = sum over bins of share * |mean f - observed rate|).
import math
import random


def s(x):
    return 1.0 / (1.0 + math.exp(-x))


def logit(p):
    return math.log(p / (1.0 - p))


def simulate(rng, n):
    rows = []
    for _ in range(n):
        z = rng.gauss(-3.3, 0.6)
        y = 1 if rng.random() < s(z) else 0
        rows.append((s(1.7 * z + 2.4), y))
    return rows


def fit(rows):
    a, b = 1.0, 0.0
    for _ in range(25):
        g = [0.0, 0.0]
        h = [0.0, 0.0, 0.0]
        ga, gb, haa, hab, hbb = [], [], [], [], []
        for f, y in rows:
            x = logit(f)
            p = s(a * x + b)
            ga.append((p - y) * x)
            gb.append(p - y)
            w = p * (1 - p)
            haa.append(w * x * x)
            hab.append(w * x)
            hbb.append(w)
        g = [math.fsum(ga), math.fsum(gb)]
        h = [math.fsum(haa), math.fsum(hab), math.fsum(hbb)]
        det = h[0] * h[2] - h[1] * h[1]
        a -= (h[2] * g[0] - h[1] * g[1]) / det
        b -= (h[0] * g[1] - h[1] * g[0]) / det
    return a, b


def ece(rows, bins=10):
    rows = sorted(rows)
    n = len(rows)
    parts, table = [], []
    for i in range(bins):
        chunk = rows[i * n // bins:(i + 1) * n // bins]
        mf = math.fsum(f for f, _ in chunk) / len(chunk)
        mo = math.fsum(y for _, y in chunk) / len(chunk)
        parts.append(len(chunk) / n * abs(mf - mo))
        table.append((mf, mo))
    return math.fsum(parts), table


def logloss(rows):
    return -math.fsum(y * math.log(f) + (1 - y) * math.log(1 - f)
                      for f, y in rows) / len(rows)


def main():
    rng = random.Random(6201)
    train, test = simulate(rng, 20000), simulate(rng, 20000)
    a, b = fit(train)
    print("Platt fit on the first half: a = {:.3f}, b = {:.3f}".format(
        a, b))
    print("ideal inverse map: a = {:.3f}, b = {:.3f}".format(
        1 / 1.7, -2.4 / 1.7))
    fixed = [(s(a * logit(f) + b), y) for f, y in test]
    e0, t0 = ece(test)
    e1, t1 = ece(fixed)
    print("second half: ECE raw {:.4f}, re-scaled {:.4f}".format(e0, e1))
    print("log loss raw {:.4f}, re-scaled {:.4f}".format(
        logloss(test), logloss(fixed)))
    print(" bin  raw mean f  observed | scaled mean f  observed")
    for i in (0, 2, 5, 7, 9):
        print("  {:>2}    {:.4f}     {:.4f}  |    {:.4f}      {:.4f}".format(
            i + 1, t0[i][0], t0[i][1], t1[i][0], t1[i][1]))


if __name__ == "__main__":
    main()

Single source and citation

Fan, Y. (2026). Next-Generation Knowledge Distillation, Context-Parametric Compilation and Live-Conversion Feedback Calibration (Version 1.0.0), Chapter 6: Calibration and the Whole Pipeline: the Brier Partition, Binned Calibration Error and a Net-Benefit Simulation. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-377 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026377,
  author    = {Fan, Yafu},
  title     = {Next-Generation Knowledge Distillation, Context-Parametric Compilation and Live-Conversion Feedback Calibration},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book