5D Library › The Algorithmic Mirror · 中文
Chapter 5 Echo Chambers and Model Collapse
When models learn from models
As generated text spreads across the internet, future models will be trained on data that partly comes from earlier models. What happens when a model is trained, generation after generation, on its own output?
Evidence
Shumailov and colleagues showed that indiscriminate training on model-generated content causes irreversible defects that they called model collapse: the tails of the original distribution disappear, and over generations the learned distribution narrows. They demonstrated it in Gaussian mixture models, variational autoencoders and language models, and argued that data from genuine human interaction will become more valuable as generated content spreads (Shumailov et al. 2024).
Evidence
Gerstgrasser and colleagues showed that the outcome depends on what happens to the original data. When each generation’s synthetic data replaces the previous data, models collapse; when the synthetic data is accumulated alongside the original real data, collapse is avoided, across language models, diffusion models and autoencoders. In a linear model they proved that with accumulation the test error stays bounded, however many generations pass (Gerstgrasser et al. 2024).
The mechanism in miniature
Calculation
Fit a Gaussian to 20 samples, draw the next 20 from the fit, and repeat for 300 generations, either replacing the data each time or keeping all earlier data, including the original real samples. The program reports the median fitted standard deviation over 20 runs.
import math, random, statistics
# Recursive training in miniature: fit a Gaussian to 20 samples, draw the next
# generation's 20 samples from the fit, and repeat. "Replace" fits only the latest
# synthetic samples; "accumulate" fits all data so far, including the original real data.
def run(mode, gens, n=20, seed=0):
rnd = random.Random(seed)
data = [rnd.gauss(0, 1) for _ in range(n)]
s, s2, m = sum(data), sum(x * x for x in data), n # running totals
sds = []
for _ in range(gens):
mu = s / m
sd = math.sqrt(max(s2 - m * mu * mu, 0) / (m - 1))
sds.append(sd)
new = [rnd.gauss(mu, sd) for _ in range(n)]
if mode == "replace":
s, s2, m = sum(new), sum(x * x for x in new), n
else:
s, s2, m = s + sum(new), s2 + sum(x * x for x in new), m + n
return sds
checkpoints = (1, 30, 100, 300)
for mode in ("replace", "accumulate"):
runs = [run(mode, 300, seed=k) for k in range(20)]
med = [statistics.median(r[g - 1] for r in runs) for g in checkpoints]
print(f"{mode:10s} median fitted SD (20 runs) at generation "
+ ", ".join(f"{g}: {v:.3f}" for g, v in zip(checkpoints, med)))
# replace median fitted SD (20 runs) at generation 1: 1.042, 30: 0.463, 100: 0.069, 300: 0.000
# accumulate median fitted SD (20 runs) at generation 1: 1.042, 30: 1.006, 100: 0.997, 300: 1.000
Each fit is slightly wrong, and the errors compound because each generation learns only from the last. Estimating a spread from a small sample tends to underestimate it slightly, and the extremes of a sample are the first things a new sample fails to reproduce, so the drift is towards narrowness. Keeping the original data anchors every fit.
Calculation
The same process on a vocabulary: 5,000 word types with Zipf frequencies, each generation learning the word frequencies of 20,000 tokens and producing the next 20,000. A word not sampled in one generation is gone for good, unless fresh real text is mixed in.
import random
# Rare words disappear first. A vocabulary of 5,000 word types with Zipf frequencies;
# each generation "learns" the empirical frequencies of 20,000 tokens and generates
# the next 20,000 tokens from them. A word that happens not to be sampled is gone for good,
# unless some fresh real text is mixed in.
random.seed(11)
V = 5000
w = [1 / (r + 1) for r in range(V)]
types = list(range(V))
def generations(fresh_share, gens=30, n=20000):
counts = None
alive = []
for g in range(gens):
n_fresh = int(n * fresh_share) if g else n
sample = random.choices(types, weights=w, k=n_fresh)
if counts and n - n_fresh:
words = list(counts)
sample += random.choices(words, weights=[counts[x] for x in words], k=n - n_fresh)
counts = {}
for t in sample:
counts[t] = counts.get(t, 0) + 1
alive.append(len(counts))
return alive
for share in (0.0, 0.1):
a = generations(share)
print(f"fresh real text {share:4.0%}: word types still in use at generations 1, 10, 30: "
f"{a[0]}, {a[9]}, {a[29]}")
# fresh real text 0%: word types still in use at generations 1, 10, 30: 3201, 1117, 559
# fresh real text 10%: word types still in use at generations 1, 10, 30: 3196, 2004, 1976Rare words disappear first, and without fresh text the vocabulary keeps shrinking. A tenth of fresh real text each generation is enough to keep the number of word types in use stable, at a level set by how much text is sampled.
Where new information comes from
The simulations, and the published results, point to a simple principle: a closed loop of generation and retraining loses information, and what keeps it alive is a supply of data from outside the loop. That data can come from many sources: new human writing, measurements, experiments, records of events, conversations, sensor data. Human experience of the physical world, and the judgement that connects distant fields, are among the most valuable of these sources, because they are hard to substitute; but they are not the only ones, and a laboratory instrument recording new measurements breaks the loop as surely as a person does.
Hypothesis · not established
Not established. In knowledge work with language models, outputs grounded in new first-hand observation (an experiment run, a measurement made, a field visit) are rated by blinded experts as more novel than outputs produced from the model and existing literature alone, at equal rigour. The hypothesis predicts higher novelty ratings, with no loss of rigour, for the first-hand condition in a randomised comparison of matched projects. It would be refuted if the novelty ratings did not differ, or were bought at the cost of rigour.
References
Fan, Y. (2026). The Algorithmic Mirror: Self-Attention Geometry, Ashby's Requisite Variety and Cognitive Projection in Information Theory and East Asian Philosophy of Mind [算法之镜] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23723557 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).