Mid-Context Forgetting in Long-Context Attention: Positional Bias and the Threshold for Lossless Segment-Summary Relays: chapter answers
Two views circulate. One says that whatever fits in the window gets read, so long material can simply be stuffed in; the other says that models reading long inputs forget the middle, so material should be cut into segments, summarised and relayed layer by layer. This book tests the founding claim of the second view: recall in the middle of a long context is markedly lower than at the ends, tracing a U (Liu et al., Transactions of the Association for Computational Linguistics, 2024), so a relay beats stuffing the window, while relay depth is bounded by the product of per-layer retention rates.…
DOI pending · monograph page goes live after publication · 5DL-B-255 · 中文版
- Chapter 1 · The Collapse in the Middle: Original Numbers, Effect Sizes and Dose for the U-Shaped Position Curve
Program p11 fits GPT-3.5-Turbo's scores at twenty documents with an upward-opening parabola 17.09 points deep. Program p12 gives the confidence interval of the 0.220 gap between beginning and middle, shows that "below c… - Chapter 2 · Dissent and Diversity: Four Curves, Bigger Windows and the Single-Needle Test
Program p21 shows the four models of the paper drawing four shapes, with only one model's middle below its closed-book score and every model carrying a clear dilution loss. Program p22 shows that extended-window version… - Chapter 3 · A Position-Recall Model: a Primacy Bump, a Recency Bump and a Floor That Sinks with Length
Program p31 fits the two-exponential model separately to thirty and twenty documents: the primacy bump is 0.249 in both, the decay lengths are 2.5 and 2.2 documents, and only the floor falls, from 0.609 to 0.578. Progra… - Chapter 4 · The Product of Retention Rates: a Ceiling on Layers, the Recency Slope of Running Summaries and the Casualties among Details
Chapter 4 computes the product of relay retention rates, shows that a running summary turns the U into a recency slope, and finds detailed facts keeping only 0.4305 after eight layers. - Chapter 5 · The Best Segment Length and the Relay Threshold: a Tug of War between Losses in Reading and in Writing
Chapter 5 solves for the best segment length and the relay threshold: for the 2023 reader each summary call must keep key facts above about 0.911 and segments must be short, and once the reader's loss falls below 0.413… - Chapter 6 · The Key-Fact Restatement Slot: Zero Loss Comes Only from Template Copying and Checksums
Chapter 6 shows that zero loss can come only from a key-fact restatement slot copied by template and verified by checksum, which lifts key-fact survival from 0.601 to above nine in ten but holds only about seven lines. - Chapter 7 · The Classics Examined: Mencius on One Day of Sun and Ten Days of Cold, and Ebbinghaus on Memory
Program p72 turns Mencius's image into parameters: with a five per cent cold-day loss, the warm day must add 0.670 to break even; in a relay, restating a key fact only every eleventh layer leaves a mean survival of 0.86… - Chapter 8 · Closing the Loop: a Relay Template for Long Tasks, the Key-Fact Slot and a Pre-Registered Test
Chapter 8 gives a relay template for long tasks, the Segment Relay Threshold Calculator and a pre-registered test. Marked boxes separate published evidence, runnable Python calculations (21 programs), classical sources…
Illustrative model calculation, not a live quote