The Minimum Description Length Principle and Ockham's Razor for Rule Bases: Two-Part Codes that Decide Whether a New Rule Earns Its Place: chapter answers
Whoever runs a rule base is tempted to believe that more rules mean better coverage. This book overturns that piece of common sense: a new rule deserves a place only if the bits it saves in encoding the exceptions exceed the bits needed to describe the rule itself. The grounds are Rissanen's principle of shortest data description, published in Automatica in 1978, and Grünwald's monograph of 2007. After checking the sources, the book corrects its blueprint: Rissanen's paper concerns statistical model selection, and its use for rule bases is later work; "saves more than it costs" is the criteri…
DOI pending · monograph page goes live after publication · 5DL-B-311 · 中文版
- Chapter 1 · Even Integers Cost Something: Universal Codes, the Kraft Inequality and the Bits of an Exception List
Program p13 compares the four at two scales, 600 records and 6,000. With 600 records and 30 exceptions, flagging costs 600 bits, listing positions 309, and the enumerative code only 177.1, just 5.25 bits above Shannon's… - Chapter 2 · The Two-Part Ledger: How Many Exceptions a Rule Must Fix to Earn Its Own Length
Chapter 2 sets up the two-part ledger: a narrow rule that is right more than eight times in ten but fixes only 7 exceptions net cannot earn its own length; on a synthetic ledger the gate admits only 3 of the proposer's… - Chapter 3 · The Overfitting Curve: Rule Count, Total Description Length and Hold-out Error
Chapter 3 draws the overfitting curve: training exceptions keep falling while hold-out error climbs back from 0.1973 to 0.2333; over twenty repetitions, the base that stops at the shortest total length has a hold-out er… - Chapter 4 · Reconciling with the Bayesian Information Criterion: Parametric Complexity, Normalised Maximum Likelihood and Where Rule Bases Part Ways
On 150 records the Bayesian information criterion admits 9.5 rules on average, with a hold-out error of 0.2512; minimum description length admits only 1.4, with an error of 0.2218; the optimum is 2.2 rules with an error… - Chapter 5 · Sensitivity to the Code: Six Reasonable Codes, Six Verdicts, and How Many Bits of Slack to Reserve
Chapter 5 measures sensitivity to the code: in twenty repetitions six reasonable codes never once choose the same rule count, every verdict with a margin below 4 bits flips with the code, and a minimum margin of 16 bits… - Chapter 6 · The Pruner's Admission Threshold: a Gate that Only Catches Repeats and a Gate that Counts Bits
Chapter 6 brings the gate to the cross-volume axiom pruner: a gate that only catches repeats admits 46 rules with a hold-out error of 0.4155, while the description-length gate admits only the 2 true rules. - Chapter 7 · Sources East and West: the Extra Finger of the Zhuangzi and Ockham's "In Vain through More"
"Lengthening the duck's legs" replaces a short rule by a longer version, which narrows a true rule. The rule half grows by only 2.0 bits, but the records narrowed away become exceptions again, the exception half grows b… - Chapter 8 · Closing the Loop: the Admission Check-up, the Rule Admission Description-Length Referee and a Pre-registered Plan of Refutation
Chapter 8 gives the admission check-up, the distilled Rule Admission Description-Length Referee and a pre-registered plan of refutation; testing whether the gate beats no gate needs 2,733 hold-out records per arm. Marke…
Illustrative model calculation, not a live quote