What does Hundred-Volume Hypergraphs and Knowledge Scaling conclude, in numbers, about “Compression and Coverage: How Much Do Fifteen Primitives Really Explain?”?
Answer: Chapter 3 finds that a shared library of primitives shortens a naive description 6.10 times but only 1.04 times against a good general compressor, and that fifteen core primitives cover 82 of the 100 volumes of a synthetic library, full coverage needing 28.
Scope of the numbers: Its synthetic library, rates and hazards are illustrative models; all calculations are generated by its programs; the book is not investment, legal, medical or technical advice. Chapter 3 finds that a shared library of primitives shortens a naive description 6.10 times but only 1.04 times against a good general compressor, and that fifteen core primitives cover 82 of the 100 volumes of a synthetic library, full coverage needing 28.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 1.04, 28, 6.10
| description raw bytes bz2 bytes lzma bytes | |
| separate | 124,389 20,291 13,864 |
| library | 20,387 13,167 13,376 |
| primitives used | 28 (15 core, 13 rare) |
| raw ratio separate / library | 6.10 |
| lzma ratio separate / library | 1.04 |
| ideal bits (5 per letter): separate 621,945, library 101,935 |
Show the full program
# Chapter 3. p06: How much does a shared library of primitives compress
# a hundred-volume description? Synthetic incidence (shared with p07 and
# p08): 100 volumes each use 2 to 5 of 15 core primitives (popularity
# falling as 1 / rank^0.8) and, with chance 0.25, one of 30 rare ones.
# Each primitive has a random definition of 300 to 580 letters from a
# 32-letter alphabet (5 bits a letter, incompressible by design); each
# use carries 24 letters of parameters. "Separate": every volume spells
# out each definition it uses. "Library": each used definition once,
# then a reference per use. Kolmogorov complexity cannot be computed; bz2
# and lzma sizes are upper bounds on it. (bz2 rather than zlib: zlib's
# output size varies between zlib builds.) Seed 100.
import bz2
import lzma
import random
ALPHA = "abcdefghijklmnopqrstuvwxyz012345"
def incidence(rng):
w = [1 / (j + 1) ** 0.8 for j in range(15)]
vols = []
for _ in range(100):
k = rng.choice((2, 2, 3, 3, 3, 4, 4, 5))
use = set()
while len(use) < k:
use.add(rng.choices(range(15), weights=w)[0])
use = sorted(use)
if rng.random() < 0.25:
use.append(15 + rng.randrange(30))
vols.append(use)
return vols
if __name__ == "__main__":
rng = random.Random(100)
vols = incidence(rng)
defs = ["".join(rng.choice(ALPHA) for _ in range(300 + 7 * j % 281))
for j in range(45)]
used = sorted({j for v in vols for j in v})
sep, ref = [], ["".join("{}:{};".format(j, defs[j]) for j in used)]
for v in vols:
for j in v:
par = "".join(rng.choice(ALPHA) for _ in range(24))
sep.append(defs[j] + par)
ref.append("#{}{}".format(j, par))
a, b = "".join(sep).encode(), "".join(ref).encode()
print(" description raw bytes bz2 bytes lzma bytes")
for name, s in (("separate", a), ("library", b)):
print(" {:<14} {:10,d} {:11,d} {:11,d}".format(
name, len(s), len(bz2.compress(s, 9)),
len(lzma.compress(s, preset=9))))
print(" primitives used: {} ({} core, {} rare)".format(
len(used), sum(j < 15 for j in used), sum(j >= 15 for j in used)))
print(" raw ratio separate / library: {:.2f}".format(len(a) / len(b)))
print(" lzma ratio separate / library: {:.2f}".format(
len(lzma.compress(a, preset=9)) / len(lzma.compress(b, preset=9))))
print(" ideal bits (5 per letter): separate {:,}, library {:,}".format(
5 * sum(len(x) for x in sep), 5 * sum(len(x) for x in ref)))Single source and citation
Fan, Y. (2026). Hundred-Volume Knowledge Hypergraphs, Multimodal Personal Prior Compilation and Autonomous Empire Scaling (Version 1.0.0), Chapter 3: Compression and Coverage: How Much Do Fifteen Primitives Really Explain?. 5D Library. https://doi.org/10.5281/zenodo.24090260
DOI: 10.5281/zenodo.24090260 · Read the monograph · 5DL-B-100 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026100,
author = {Fan, Yafu},
title = {Hundred-Volume Knowledge Hypergraphs, Multimodal Personal Prior Compilation and Autonomous Empire Scaling},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.24090260},
url = {https://doi.org/10.5281/zenodo.24090260}
}Other chapter answers in this book
- Chapter 1 · One Equation, Many Worlds: What a Cross-Disciplinary Isomorphism Is and Is Not
- Chapter 2 · Three More Primitives: Weibull Hazards, Legendre-Bregman Duality and Bayesian Updating
- Chapter 4 · The Library as a Hypergraph: Degrees, Co-Use and Connected Volumes
- Chapter 5 · The Feedback Flywheel: Fisher Information, Privacy Noise and Self-Confirming Routing
- Chapter 6 · A Hundred Years: Copies, Common Shocks and What Reproducibility Can Promise
- Chapter 7 · "Store It in a Famous Mountain": Sima Qian, the Library of Alexandria and the *Taiping yulan*
- Chapter 8 · Practice: Zero Marginal Cost, Rising Consistency Cost and a Fixity Schedule