What does Distilling Books into Skill Packs conclude, in numbers, about “Extraction: Picking the Executable Sentences out of a Manuscript One by One”?
Answer: Rule sentences come in eight patterns: condition–action, threshold, prohibition, ordering, exception, formula, refusal and definition; the annotation box is a ninth signal. On the 30-sentence fictional quotation handbook, the box-aware cue extractor reaches precision 0.8966 and recall 0.8387 in Chinese, precision 0.8333 and recall 0.9677 in English; box signals remove false positives at no cost in recall, and the bilingual union misses only one implicit conditional that carries no cue at all. Chinese quantity words must be converted first, and those that cannot be converted must be refused. Conclusions about recall are bounded by the number of gold rules: a target of 0.90 with no misses needs at least 35. Kappa between two annotators is driven by the share of rule sentences, so compare that share before comparing manuscripts. The product of extraction is an extraction sheet with every field filled, and Chapter 3 starts from that sheet to compile rule sentences into executable predicates.
Scope of the numbers: All samples are invented; pricing and closing rules are written in general terms and are not legal or compliance advice.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.8333, 0.8387, 0.8966, 0.90, 0.9677, 30
| recall with 95% Wilson interval | |
| zh overall | 26/31 = 0.8387 [0.6737, 0.9291] |
| en overall | 30/31 = 0.9677 [0.8381, 0.9943] |
| zh COND | 3/ 6 = 0.5000 [0.1876, 0.8124] |
| zh THRESH | 6/ 7 = 0.8571 [0.4869, 0.9743] |
| PROHIB | 3/ 3 = 1.0000 [0.4385, 1.0000] |
| EXCEPT | 2/ 2 = 1.0000 [0.3424, 1.0000] |
| zh recall: micro 0.8387, macro over 8 types 0.8780 | |
| en recall: micro 0.9677, macro over 8 types 0.9792 | |
| zh: P 0.8966 R 0.8387 F1 0.8667 F2 0.8497 F0.5 0.8844 | |
| en: P 0.8333 R 0.9677 F1 0.8955 F2 0.9375 F0.5 0.8571 | |
| gold rules needed so that the Wilson lower bound on recall reaches the target | |
| target 0.80 | 0 miss -> n 16; 1 miss -> n 25; 2 miss -> n 33; 3 miss -> n 40 |
| target 0.90 | 0 miss -> n 35; 1 miss -> n 53; 2 miss -> n 69; 3 miss -> n 84 |
| target 0.95 | 0 miss -> n 73; 1 miss -> n 110; 2 miss -> n 142; 3 miss -> n 173 |
| zero misses in n gold rules: one-sided 95% upper bound on miss rate | 1 - 0.05^(1/n) |
| n | 10: exact 0.2589, rule of three 3/n 0.3000 |
| n | 30: exact 0.0950, rule of three 3/n 0.1000 |
| n 100: exact 0.0295, rule of three 3/n 0.0300 | |
| n 300: exact 0.0099, rule of three 3/n 0.0100 |
Show the full program
# Chapter 2. p21: Wilson intervals for extraction recall, micro vs macro averages, and audit sizes.
# Counts are the box-aware results printed by p20 (TP, FN per type; TP, FP overall).
# Wilson score interval: centre (k + z^2/2) / (n + z^2), half-width z sqrt(k(n-k)/n + z^2/4) / (n + z^2).
# F-beta = (1 + b^2) P R / (b^2 P + R); b = 2 weighs a missed rule four times a false one.
import math
from fractions import Fraction
Z = 1.959964 # two-sided 95 %
def wilson(k, n, z=Z):
c = (k + z * z / 2) / (n + z * z)
h = z * math.sqrt(k * (n - k) / n + z * z / 4) / (n + z * z)
return c - h, c + h
def fbeta(p, r, b):
return (1 + b * b) * p * r / (b * b * p + r)
# per type (TP, FN) from p20, box-aware
ZH = {"COND": (3, 3), "THRESH": (6, 1), "PROHIB": (3, 0), "ORDER": (4, 0), "EXCEPT": (2, 0),
"FORMULA": (3, 0), "REFUSE": (3, 0), "DEF": (2, 1)}
EN = {"COND": (5, 1), "THRESH": (7, 0), "PROHIB": (3, 0), "ORDER": (4, 0), "EXCEPT": (2, 0),
"FORMULA": (3, 0), "REFUSE": (3, 0), "DEF": (3, 0)}
OVERALL = {"zh": (26, 3, 5), "en": (30, 6, 1)} # TP, FP, FN
if __name__ == "__main__":
print(" recall with 95% Wilson interval")
for name, (k, n) in [("zh overall", (26, 31)), ("en overall", (30, 31)), ("zh COND", (3, 6)),
("zh THRESH", (6, 7)), ("PROHIB", (3, 3)), ("EXCEPT", (2, 2))]:
lo, hi = wilson(k, n)
print(" {:11s} {:2d}/{:2d} = {:.4f} [{:.4f}, {:.4f}]".format(name, k, n, k / n, lo, hi))
for lang, table in (("zh", ZH), ("en", EN)):
tp = sum(a for a, _ in table.values())
fn = sum(b for _, b in table.values())
micro = Fraction(tp, tp + fn)
macro = sum(Fraction(a, a + b) for a, b in table.values()) / len(table)
print(" {} recall: micro {:.4f}, macro over 8 types {:.4f}".format(
lang, float(micro), float(macro)))
for lang, (tp, fp, fn) in OVERALL.items():
p, r = tp / (tp + fp), tp / (tp + fn)
print(" {}: P {:.4f} R {:.4f} F1 {:.4f} F2 {:.4f} F0.5 {:.4f}".format(
lang, p, r, fbeta(p, r, 1), fbeta(p, r, 2), fbeta(p, r, 0.5)))
print(" gold rules needed so that the Wilson lower bound on recall reaches the target")
for target in (0.80, 0.90, 0.95):
row = []
for misses in (0, 1, 2, 3):
n = misses + 1
while wilson(n - misses, n)[0] < target:
n += 1
row.append("{} miss -> n {}".format(misses, n))
print(" target {:.2f}: {}".format(target, "; ".join(row)))
print(" zero misses in n gold rules: one-sided 95% upper bound on miss rate = 1 - 0.05^(1/n)")
for n in (10, 30, 100, 300):
print(" n {:3d}: exact {:.4f}, rule of three 3/n {:.4f}".format(n, 1 - 0.05 ** (1 / n), 3 / n))Single source and citation
Fan, Y. (2026). Lossless Distillation of Monographs into Agent Skill Packs, Executable Rule Compilation and Self-Evolving Systems (Version 0.1.0), Chapter 2: Extraction: Picking the Executable Sentences out of a Manuscript One by One. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-150 · v0.1.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026150,
author = {Fan, Yafu},
title = {Lossless Distillation of Monographs into Agent Skill Packs, Executable Rule Compilation and Self-Evolving Systems},
year = {2026},
publisher = {5D Library},
edition = {0.1.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Method: Skill Packs, Four Retention Indicators, Evidence Grades and the Four Steps of Distillation
- Chapter 3 · Compilation: Turning Rule Sentences into Predicates, Decision Tables and Decision Trees
- Chapter 4 · Guardrails and Validators: Banned Phrases, Required Sentences, Refusal Conditions, Lint and Golden Samples
- Chapter 5 · One Specification for the Four Kits: Pricing Calculator, Front-End Style Library, Persuasive-Copy Checker and Customer-Service Closing Pack
- Chapter 6 · Self-Evolution: Feedback Loops, Regression Gates, Drift and the Accumulation of a Skill Library
- Chapter 7 · Governance: Least Privilege, Machine-Checked Red Lines, Disclaimers, Provenance and Rule Retirement
- Chapter 8 · Practice: Distilling a Twelve-Rule Pricing Book into a Pricing Calculator