What does Distilling Books into Skill Packs conclude, in numbers, about “Method: Skill Packs, Four Retention Indicators, Evidence Grades and the Four Steps of Distillation”?

Answer: Chapter 1 establishes two counter-intuitive results: pasting a whole book into the prompt gives coverage 1.000 but effective retention 0.000; with twelve samples all passing, the 95% upper bound on the failure rate is still 0.2209, and bringing it below one percent takes 299; under stated assumptions, a twelve-rule pack without samples loses at least one rule within ten edits with probability 0.9115.

Scope of the numbers: All samples are invented; pricing and closing rules are written in general terms and are not legal or compliance advice.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.2209, 0.9115, 299, 95

golden n all pass95% upper bound on failure rate 3/n
120.2209 0.2500
240.1173 0.1250
500.0582 0.0600
1000.0295 0.0300
2990.0100 0.0100
3000.0099 0.0100
samples needed so that zero failures bound the rate below 0.0559
samples needed so that zero failures bound the rate below 0.01299
q0.02 per rule per round; share of rules intact after n rounds
coverage c n1 n=5 n=10 n=20 P(any of 12 broken by n=10)
0.00.9800 0.9039 0.8171 0.6676 0.9115
0.50.9900 0.9510 0.9044 0.8179 0.7006
0.90.9980 0.9900 0.9802 0.9608 0.2136
1.01.0000 1.0000 1.0000 1.0000 0.0000
with no tests, half of the rules are gone after 34.3 rounds
Show the full program
# Chapter 1. p12: What "all golden samples pass" proves, and how fast an untested pack decays (closed form).
# Part 1, zero numerator: if n independent golden samples all pass, the one-sided 95% upper bound on
# the per-case failure rate is 1 - 0.05 ** (1 / n), close to 3 / n (the "rule of three").
# Part 2, silent decay (a model, parameters assumed, not measured): each edit round breaks each rule
# with probability q = 0.02, independently; a golden suite notices a broken rule with probability c
# and the edit is reverted. A rule survives one round with probability 1 - q * (1 - c), so after
# n rounds the expected share of rules intact is (1 - q * (1 - c)) ** n. For a 12-rule pack the chance
# that at least one rule is broken is 1 - (1 - q * (1 - c)) ** (12 * n). Chapter 6 simulates this.
from math import ceil, log

Q = 0.02
RULES = 12


def upper95(n):
    return 1 - 0.05 ** (1 / n)


if __name__ == "__main__":
    print("   golden n   all pass -> 95% upper bound on failure rate   3/n")
    for n in (12, 24, 50, 100, 299, 300):
        print("   {:8d}   {:44.4f}   {:.4f}".format(n, upper95(n), 3 / n))
    for target in (0.05, 0.01):
        need = ceil(log(0.05) / log(1 - target))
        print("   samples needed so that zero failures bound the rate below {:.2f}: {}".format(target, need))
    print("   q = {:.2f} per rule per round; share of rules intact after n rounds".format(Q))
    print("   coverage c     n=1     n=5    n=10    n=20   P(any of 12 broken by n=10)")
    for c in (0.0, 0.5, 0.9, 1.0):
        s = 1 - Q * (1 - c)
        row = ["{:7.4f}".format(s ** n) for n in (1, 5, 10, 20)]
        print("   {:10.1f} {}   {:25.4f}".format(c, " ".join(row), 1 - s ** (RULES * 10)))
    half = log(0.5) / log(1 - Q)
    print("   with no tests, half of the rules are gone after {:.1f} rounds".format(half))

Single source and citation

Fan, Y. (2026). Lossless Distillation of Monographs into Agent Skill Packs, Executable Rule Compilation and Self-Evolving Systems (Version 0.1.0), Chapter 1: Method: Skill Packs, Four Retention Indicators, Evidence Grades and the Four Steps of Distillation. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-150 · v0.1.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026150,
  author    = {Fan, Yafu},
  title     = {Lossless Distillation of Monographs into Agent Skill Packs, Executable Rule Compilation and Self-Evolving Systems},
  year      = {2026},
  publisher = {5D Library},
  edition   = {0.1.0},
  note      = {DOI pending}
}

Other chapter answers in this book