What does Anti-Collapse Engineering with Two Anchors conclude, in numbers, about “Examining for Collapse: Why Inward-Looking Metrics Go Blind, and How Much Data an Outward-Looking Anchor Needs”?
Answer: Chapter 5 shows that inward-looking detection metrics grow more optimistic during collapse, the Good–Turing missing mass falling from 0.0677 to 0.0009 while the true loss rises.
Scope of the numbers: Parameters are illustrative; all calculations are generated by the book's programs, which call no model interface and use no reader's or customer's data. Marked boxes separate published evidence, runnable Python calculations (22 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0009, 0.0677
| true support 500 | ||||
| gen | S_obs | Chao1 | GT_missing | true_missing |
| 0 | 360.9 | 467.9 | 0.0677 | 0.0661 |
| 1 | 298.2 | 351.0 | 0.0405 | 0.1090 |
| 2 | 255.1 | 288.5 | 0.0281 | 0.1425 |
| 5 | 187.2 | 200.5 | 0.0128 | 0.2095 |
| 10 | 132.4 | 137.3 | 0.0052 | 0.2861 |
| 20 | 90.6 | 95.3 | 0.0030 | 0.3591 |
| 50 | 47.0 | 47.6 | 0.0009 | 0.4792 |
Show the full program
# Chapter 5. p51: inward-looking detectors cannot see a lost tail.
# Model: K = 500 categories, Zipf p_i proportional to 1/i, N = 2000
# items per generation, each generation refitted on the counts of the
# one before (as in p23). On each generation's own sample we compute
# the Good-Turing missing mass f1/N (Good 1953) and the bias-corrected
# Chao1 richness S_obs + f1(f1-1)/(2(f2+1)) (Chao 1984), and compare
# them with the truth: the true mass of categories absent from the
# sample. R = 10 chains, seed fixed; means are shown.
import collections
import itertools
import math
import random
K, N, R = 500, 2000, 10
CHECK = (0, 1, 2, 5, 10, 20, 50)
H = math.fsum(1.0 / i for i in range(1, K + 1))
P = [1.0 / (i * H) for i in range(1, K + 1)]
rng = random.Random(20261114)
cats = list(range(K))
acc = collections.defaultdict(list)
for _ in range(R):
cum = list(itertools.accumulate(P))
for gen in range(max(CHECK) + 1):
counts = collections.Counter(rng.choices(cats, cum_weights=cum,
k=N))
if gen in CHECK:
freq = collections.Counter(counts.values())
f1, f2 = freq[1], freq[2]
s_obs = len(counts)
chao = s_obs + f1 * (f1 - 1) / (2.0 * (f2 + 1))
miss = math.fsum(P[k] for k in cats if counts[k] == 0)
acc[gen].append((s_obs, chao, f1 / N, miss))
cum = list(itertools.accumulate(counts[k] for k in cats))
print("true support %d" % K)
print("gen S_obs Chao1 GT_missing true_missing")
for gen in CHECK:
m = [math.fsum(c) / R for c in zip(*acc[gen])]
print("%-4d %-7.1f %-7.1f %-11.4f %.4f" % (gen, m[0], m[1], m[2],
m[3]))Single source and citation
Fan, Y. (2026). Anti-Collapse Engineering with Two Anchors of Truth: the Variance-Collapse Rate of Self-Generated Training Data and Immunity through Standard-Library Recomputation and Textual Collation (Version 1.0.0), Chapter 5: Examining for Collapse: Why Inward-Looking Metrics Go Blind, and How Much Data an Outward-Looking Anchor Needs. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-265 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026265,
author = {Fan, Yafu},
title = {Anti-Collapse Engineering with Two Anchors of Truth: the Variance-Collapse Rate of Self-Generated Training Data and Immunity through Standard-Library Recomputation and Textual Collation},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Variance Collapsing Generation by Generation: the Expectation Formula of a Gaussian Feedback Chain and Almost-Sure Extinction
- Chapter 2 · The Tails Die First: Generational Extinction in a Discrete Law and the Absorbing States of Wright–Fisher
- Chapter 3 · Replace or Accumulate: the Counter-Evidence of Gerstgrasser and the Bound of pi^2/6
- Chapter 4 · The Threshold of Mixing in Real Sources: How Much Variance a Fixed Share Preserves, and How Much of the Tail It Misses
- Chapter 6 · The First Anchor: Standard-Library Recomputation, Deterministic Fingerprints and the Correlated Failure of Two Anchors
- Chapter 7 · The Classics Examined: Duan Yucai on "the Difficulty of Settling Right and Wrong" and Erasmus's Return to the Original in 1516
- Chapter 8 · The Empirical Loop: a Two-Anchor Procedure for Every Volume, the Feedback Collapse Rate Calculator and a Pre-Registered Test