5D Library › Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades · 中文

Chapter 3. The Context Window: Position Decay, Evidence Dilution and the Best Number of Chunks

Does a bigger window answer better?

Long-context models tempt one to pack as many retrieved chunks into the window as possible: put in a few more, and the right one is more likely to be among them. Liu and colleagues’ experiments showed the other side: models use information at the beginning and end of the context best and in the middle worst, a U-shape; with the relevant information buried in the middle, accuracy can fall below that with no documents at all (Liu et al. 2024). This chapter puts the two opposing forces — rising recall and diluted evidence — into one stylised reader model and watches how accuracy changes with the number of chunks.

Calculation

A stylised reader model. Position pp gets attention weight w(p)=0.25+0.75max⁡(e−p/3,e−(K−1−p)/3)w(p) = 0.25 + 0.75\max(e^{-p/3}, e^{-(K-1-p)/3}), high at both ends and low in the middle. The gold chunk pulls with strength 1, each distractor with ss, and the model’s own (possibly wrong) prior with 0.2. When the gold chunk is present, the probability of a correct answer follows Luce’s choice rule (Luce 1959): wgoldwgold+s∑wdistractor+0.2,\frac{w_{\text{gold}}}{w_{\text{gold}} + s\sum w_{\text{distractor}} + 0.2} , and 0.15 when it is absent. The retriever ranks the gold chunk first with probability 0.35 and otherwise further down with a geometric tail. This is a model to show the mechanism, not a measurement of any particular language model.

Calculation

# Chapter 3. p06: More context can mean less answer: position and dilution.
# A stylised reader model, not a measured one. The context holds K chunks
# in the retriever's order. Each position p (0 = first) gets attention
# weight w(p) = 0.25 + 0.75 max(exp(-p/3), exp(-(K-1-p)/3)) - high at both
# ends, low in the middle, the U-shape reported by Liu et al. (2024). The
# gold chunk pulls with strength 1, every distractor with strength s, and
# the model's own (possibly wrong) prior with 0.2. Probability of a
# correct answer when the gold chunk is present (Luce choice rule):
#     w_gold / (w_gold + s sum w_distractors + 0.2)
# and 0.15 when it is absent. The retriever puts the gold chunk at rank
# 1 with probability 0.35, otherwise further down with a geometric tail:
# recall@K = 1 - 0.65 * 0.8^(K - 1). Two distractor strengths: s = 0.03
# (clearly off-topic chunks) and s = 0.25 (plausible near-misses).
import math

def w(p, k):
    return 0.25 + 0.75 * max(math.exp(-p / 3), math.exp(-(k - 1 - p) / 3))

def correct(pos, k, s):
    ws = [w(p, k) for p in range(k)]
    g = ws[pos]
    return g / (g + s * (sum(ws) - g) + 0.2)

def p_rank(r):
    return 0.35 if r == 1 else 0.65 * 0.2 * 0.8 ** (r - 2)

def expected(k, s):
    acc = sum(p_rank(r) * correct(r - 1, k, s) for r in range(1, k + 1))
    miss = 1 - sum(p_rank(r) for r in range(1, k + 1))
    return acc + miss * 0.15, 1 - miss

if __name__ == "__main__":
    print("A. K = 20 chunks, s = 0.25, gold chunk placed at position p")
    for p in (0, 2, 5, 10, 15, 19):
        print("   position {:2d}: P(correct) {:.3f}".format(
            p, correct(p, 20, 0.25)))
    print("\nB. retriever order, K chunks in the context")
    print("     K   recall@K   P(correct), s = 0.03   P(correct), s = 0.25")
    best = {0.03: (0, 0), 0.25: (0, 0)}
    for k in (1, 2, 3, 5, 8, 10, 15, 20, 40):
        row = []
        for s in (0.03, 0.25):
            a, rec = expected(k, s)
            best[s] = max(best[s], (a, k))
            row.append(a)
        print("   {:3d}    {:6.3f}          {:6.3f}                {:6.3f}"
              .format(k, rec, row[0], row[1]))
    for s in (0.03, 0.25):
        print("   s = {}: best K = {} (P(correct) {:.3f})".format(
            s, best[s][1], best[s][0]))
    print("\nReading: the gold chunk is worth most at either end of the")
    print("context and least in the middle. Adding chunks raises recall,")
    print("but each distractor dilutes the gold chunk's share. With")
    print("off-topic distractors accuracy keeps rising for a while; with")
    print("plausible near-misses it peaks at very few chunks and then")
    print("falls while recall is still climbing. The best K is a property")
    print("of the corpus and the reader, not a constant.")
# A. K = 20 chunks, s = 0.25, gold chunk placed at position p
#    position  0: P(correct) 0.288
#    position  2: P(correct) 0.198
#    position  5: P(correct) 0.130
#    position 10: P(correct) 0.098
#    position 15: P(correct) 0.146
#    position 19: P(correct) 0.288
#
# B. retriever order, K chunks in the context
#      K   recall@K   P(correct), s = 0.03   P(correct), s = 0.25
#      1     0.350           0.389                 0.389
#      2     0.480           0.468                 0.409
#      3     0.584           0.522                 0.407
#      5     0.734           0.585                 0.377
#      8     0.864           0.614                 0.323
#     10     0.913           0.612                 0.293
#     15     0.971           0.587                 0.240
#     20     0.991           0.561                 0.208
#     40     1.000           0.490                 0.147
#    s = 0.03: best K = 8 (P(correct) 0.614)
#    s = 0.25: best K = 2 (P(correct) 0.409)
#
# Reading: the gold chunk is worth most at either end of the
# context and least in the middle. Adding chunks raises recall,
# but each distractor dilutes the gold chunk's share. With
# off-topic distractors accuracy keeps rising for a while; with
# plausible near-misses it peaks at very few chunks and then
# falls while recall is still climbing. The best K is a property
# of the corpus and the reader, not a constant.

Part A places the gold chunk at different positions in a 20-chunk context: at the beginning or end the probability of a correct answer is 0.288, in the very middle (position 10) only 0.098 — the U-shape. Part B fills the context with KK chunks in retrieval order. Recall rises steadily with KK: 0.350 at K=1K = 1, 0.913 at K=10K = 10, 1.000 at K=40K = 40. Accuracy does not: with clearly off-topic distractors (s=0.03s = 0.03) it rises and then falls, peaking at K=8K = 8 (0.614) and down to 0.490 at K=40K = 40; with plausible near-misses (s=0.25s = 0.25) the best KK is just 2 (0.409), and by K=20K = 20 accuracy is down to 0.208 while recall stands at 0.991.

So “packing in too many low-relevance chunks dilutes the right evidence” holds, and the best number of chunks is not a constant: it depends on how much the distractors resemble the gold chunk (a property of the corpus) and on how sensitive the reader is to position (a property of the model).

Probability of a correct answer (solid) and recall (dashed) against the number of chunks K, for two distractor strengths; recall keeps rising while accuracy rises and then falls.

Put the best chunks at both ends

Calculation

# Chapter 3. p07: Where to put the best chunks.
# Reader model and retriever of p06 (U-shaped position weights, gold
# chunk at rank 1 with probability 0.35, geometric tail). With K chunks
# fixed, three ways to lay out the retriever's ranks in the context:
#   forward   rank 1 first, rank K last
#   reverse   rank K first, rank 1 last (nearest the question)
#   edges     ranks 1, 3, 5, ... from the front and 2, 4, 6, ... from the
#             back, so the strongest chunks sit at both ends
import math

def w(p, k):
    return 0.25 + 0.75 * max(math.exp(-p / 3), math.exp(-(k - 1 - p) / 3))

def correct(pos, k, s):
    ws = [w(p, k) for p in range(k)]
    g = ws[pos]
    return g / (g + s * (sum(ws) - g) + 0.2)

def p_rank(r):
    return 0.35 if r == 1 else 0.65 * 0.2 * 0.8 ** (r - 2)

def layout(name, k):
    if name == "forward":
        return list(range(k))
    if name == "reverse":
        return list(range(k - 1, -1, -1))
    pos, front, back = [0] * k, 0, k - 1
    for r in range(k):
        if r % 2 == 0:
            pos[r], front = front, front + 1
        else:
            pos[r], back = back, back - 1
    return pos

def expected(k, s, name):
    pos = layout(name, k)
    acc = sum(p_rank(r + 1) * correct(pos[r], k, s) for r in range(k))
    miss = 1 - sum(p_rank(r) for r in range(1, k + 1))
    return acc + miss * 0.15

if __name__ == "__main__":
    print("   K    s      forward   reverse   edges    edges - forward")
    for k in (5, 10, 20):
        for s in (0.03, 0.25):
            f, r, e = (expected(k, s, n) for n in
                       ("forward", "reverse", "edges"))
            print("  {:3d}  {:4.2f}    {:.3f}     {:.3f}     {:.3f}"
                  "     {:+.3f}".format(k, s, f, r, e, e - f))
    print("\nReading: because the reader favours both ends, putting the")
    print("strongest chunks at both ends recovers part of what the middle")
    print("loses. The gain grows with K, since a longer context has more")
    print("middle; it cannot exceed what the retriever put in the list.")
    print("Forward and reverse tie only because this model's U is")
    print("symmetric; a real reader that favours the end nearest the")
    print("question would separate them.")
#    K    s      forward   reverse   edges    edges - forward
#     5  0.03    0.585     0.585     0.590     +0.005
#     5  0.25    0.377     0.377     0.384     +0.007
#    10  0.03    0.612     0.612     0.634     +0.022
#    10  0.25    0.293     0.293     0.313     +0.020
#    20  0.03    0.561     0.561     0.607     +0.047
#    20  0.25    0.208     0.208     0.237     +0.029
#
# Reading: because the reader favours both ends, putting the
# strongest chunks at both ends recovers part of what the middle
# loses. The gain grows with K, since a longer context has more
# middle; it cannot exceed what the retriever put in the list.
# Forward and reverse tie only because this model's U is
# symmetric; a real reader that favours the end nearest the
# question would separate them.

Since the reader favours both ends, the highest-scoring chunks can be placed alternately at the beginning and the end (“edges” layout), with the weaker ones in the middle. Against placing them in score order, the gain grows with KK: only 0.005 to 0.007 at K=5K = 5, but 0.047 at K=20K = 20 with off-topic distractors. Forward and reverse order tie exactly in this model only because its U is symmetric; real models often favour the end nearest the question, which would separate them.

Note

Reordering allocates; it cannot create. Reordering changes where the gold chunk lands, not whether it was retrieved. A chunk that was not retrieved helps nowhere; before choosing KK and a layout, look at where the retriever ranks the gold chunk.

Hypothesis · not established

The author’s hypothesis (not yet tested): for a given language model and corpus, the gain in question-answering accuracy from the “edges” layout over score order grows with the number of chunks KK, and is under 1 percentage point for K≤5K \le 5. What would show it wrong: on a pre-registered question set, the gain at K=20K = 20 is no larger than at K=5K = 5, or the gain at K≤5K \le 5 exceeds 1 point.

Summary

The reader uses the ends of the context better than the middle. More chunks raise recall, but each distractor dilutes the gold chunk’s share: with off-topic distractors accuracy peaks after a handful of chunks, with plausible near-misses at about two, after which recall keeps rising while accuracy falls. Putting the strongest chunks at both ends recovers part of the middle’s loss, more so the more chunks there are; but reordering cannot make up for a chunk that was never retrieved.

References

Liu, Nelson F., Kevin Lin, John Hewitt, et al. 2024. “Lost in the Middle: How Language Models Use Long Contexts.” Transactions of the Association for Computational Linguistics 12: 157–73.
Luce, R. Duncan. 1959. Individual Choice Behavior: A Theoretical Analysis. Wiley.

Fan, Y. (2026). Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades: High-Dimensional Concentration of Measure, Hybrid Lexical-Dense Retrieval Bounds and Verifier-Gated Fixed Points [检索增强生成几何、倒数秩融合与多智能体误差级联] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090170 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).