5D Library › Spiritual Quotient Quantification, Prior-Conditioned Generation and Retrieval-Augmented Cognitive Agents · 中文

Chapter 4. The Context Budget: Lagrangian Water-Filling and the Best Number of Passages

Three parties share one window

A context window is finite. Each answer must share its tokens among three things: the user profile (P), retrieved passages (R) and conversation history (H). Each has diminishing returns: past a certain profile length, number of passages or depth of history, more adds little. The program writes the returns as Ui=ailn⁡(1+ti/si)U_i = a_i \ln(1 + t_i/s_i) with weights a=(1.0,3.0,1.5)a = (1.0, 3.0, 1.5) and half-saturation lengths s=(300,2000,1500)s = (300, 2000, 1500) tokens; the profile cannot exceed its 8 KB file, about 2,048 tokens; and a further 2,048 tokens of the window are kept for the answer.

Maximising ∑iUi\sum_i U_i under a total budget BB, the Lagrange conditions give the “water-filling” solution ti=min⁡(capi,max⁡(0,ai/ν−si))t_i = \min(\mathrm{cap}_i, \max(0, a_i/\nu – s_i)), where the multiplier ν\nu is the marginal value of the last token, found by bisection (Boyd and Vandenberghe 2004).

Calculation

# Chapter 4. p07: Splitting a context window: a Lagrangian water-filling.
# A context of T tokens, less 2,048 kept for the answer, is shared by
# the profile (P), retrieved passages (R) and conversation history (H).
# Each part has diminishing returns, U_i = a_i ln(1 + t_i / s_i), with
# weights a = (1.0, 3.0, 1.5) and half-saturation sizes s = (300,
# 2,000, 1,500) tokens; the profile cannot exceed its 8 KB file, about
# 2,048 tokens. Maximising sum U_i subject to sum t_i = B gives the
# Karush-Kuhn-Tucker solution t_i = clip(a_i / nu - s_i, 0, cap_i), with
# the multiplier nu (the value of one more token) found by bisection.
# Compared with an equal three-way split (the profile's surplus over
# its cap shared by the other two).
import math

A = (1.0, 3.0, 1.5)
S = (300.0, 2000.0, 1500.0)
CAP = (2048.0, math.inf, math.inf)
NAMES = ("profile", "retrieval", "history")

def alloc(nu):
    return [min(c, max(0.0, a / nu - s)) for a, s, c in zip(A, S, CAP)]

def solve(budget):
    lo, hi = 1e-9, 10.0
    for _ in range(200):
        mid = math.sqrt(lo * hi)
        if sum(alloc(mid)) > budget:
            lo = mid
        else:
            hi = mid
    return alloc(hi), hi

def utility(t):
    return sum(a * math.log(1 + x / s) for a, s, x in zip(A, S, t))

if __name__ == "__main__":
    print("   window   budget  profile  retrieval  history"
          "    nu x 1e4  gain vs equal")
    for T in (8192, 32768, 131072):
        b = T - 2048
        t, nu = solve(b)
        p0 = min(b / 3, CAP[0])
        eq = [p0, (b - p0) / 2, (b - p0) / 2]
        print("   {:6d} {:8d} {:8.0f} {:10.0f} {:8.0f} {:11.3f} {:13.4f}"
              .format(T, b, t[0], t[1], t[2], 1e4 * nu,
                      utility(t) - utility(eq)))
    print("\nReading: at 8k the profile already takes a quarter of the")
    print("budget; from 32k it sits at its 8 KB cap, and every further")
    print("token goes to retrieval and history in the ratio of their")
    print("weights. A larger window makes the profile's share shrink,")
    print("and the value of the last token (nu) falls sixteen-fold.")
#    window   budget  profile  retrieval  history    nu x 1e4  gain vs equal
#      8192     6144     1508       3424     1212       5.531        0.2134
#     32768    30720     2048      19448     9224       1.399        0.2321
#    131072   129024     2048      84984    41992       0.345        0.2491
#
# Reading: at 8k the profile already takes a quarter of the
# budget; from 32k it sits at its 8 KB cap, and every further
# token goes to retrieval and history in the ratio of their
# weights. A larger window makes the profile's share shrink,
# and the value of the last token (nu) falls sixteen-fold.
Shares of profile, retrieval and history in the budget as the window grows. Log scale, base 2.

In an 8k window (budget 6144) the profile gets 1508 tokens, already a quarter of the budget; from 32k the profile sits at its cap of 2,048 and everything else goes to retrieval and history by weight: in a 128k window, 84984 to retrieval and 41992 to history. The larger the window, the smaller the profile’s share, and the marginal value of the last token ν\nu falls from 5.531 to 0.345 (in units of 10−410^{-4}), sixteen-fold. Against an equal three-way split, the optimal division gains between 0.2134 and 0.2491 units across the windows.

In other words, “a larger window lets the personal profile do more” does not hold; a larger window mainly benefits retrieval and history.

How many passages to retrieve?

More passages are not always better. Language models make poorer use of information in the middle of long contexts (Liu et al. 2024), and extra passages distract. The program lets the needed passage appear in the top kk with probability 1−e−k/κ1 – e^{-k/\kappa} (the smaller κ\kappa, the better the retriever); with the passage present the model answers correctly with probability 0.90−0.01(k−1)0.90 – 0.01(k-1), without it from its prior with probability 0.30; each passage also has a cost cc.

Calculation

# Chapter 4. p08: How many passages to retrieve: recall against distraction.
# The passage that answers a question is among the top k retrieved with
# probability recall(k) = 1 - exp(-k / kappa); kappa measures how good
# the retriever is (small = good). With the passage present the model
# answers correctly with probability 0.90 - 0.01 (k - 1) (each extra
# passage distracts a little); without it, from its prior, 0.30. Each
# 512-token passage also costs c in accuracy-equivalent units (latency,
# money): c = 0 or 0.002. The program finds the k from 1 to 40 that
# maximises accuracy - c k, for kappa = 2, 4 and 8.
import math

def acc(k, kappa):
    rec = 1 - math.exp(-k / kappa)
    return rec * (0.90 - 0.01 * (k - 1)) + (1 - rec) * 0.30

if __name__ == "__main__":
    print("   kappa   cost   best k   accuracy   recall at best k")
    for kappa in (2, 4, 8):
        for c in (0.0, 0.002):
            k = max(range(1, 41), key=lambda j: acc(j, kappa) - c * j)
            print("   {:5d} {:6.3f} {:8d} {:10.4f} {:18.4f}".format(
                kappa, c, k, acc(k, kappa), 1 - math.exp(-k / kappa)))
    print("\nReading: more passages are not always better. Distraction")
    print("caps the useful k at a handful for a good retriever and at")
    print("about twice that for a weak one; a weak retriever cannot be")
    print("rescued by a larger window, only by better ranking.")
#    kappa   cost   best k   accuracy   recall at best k
#        2  0.000        7     0.8237             0.9698
#        2  0.002        6     0.8226             0.9502
#        4  0.000       10     0.7681             0.9179
#        4  0.002       10     0.7681             0.9179
#        8  0.000       15     0.6895             0.8466
#        8  0.002       14     0.6883             0.8262
#
# Reading: more passages are not always better. Distraction
# caps the useful k at a handful for a good retriever and at
# about twice that for a weak one; a weak retriever cannot be
# rescued by a larger window, only by better ranking.

A good retriever (κ=2\kappa = 2) does best with only 7 passages, at accuracy 0.8237; a middling one (κ=4\kappa = 4) with 10, at 0.7681; a poor one (κ=8\kappa = 8) with 15, at only 0.6895. A poor retriever cannot be rescued by a larger window, only by better ranking.

Hypothesis · not established

The author’s interpretation (not yet tested): in the author’s system, limiting the profile to a summary of about 2,048 tokens and retrieving 6 to 10 passages is more accurate than putting the full profile and many passages into a long window. What would show this wrong: if, in the blind comparison, the long-window, full-profile version won clearly on user preference, the diminishing-returns assumption of this chapter would not hold for real models.

Summary

Under diminishing returns, the optimal division of the context is a water-filling solution: from 32k the profile stays at its 8 KB cap, and the rest goes to retrieval and history by weight. The best number of passages is set jointly by retrieval quality and distraction: a good retriever needs only 7, and a poor one cannot catch up however many it takes.

References

Boyd, Stephen, and Lieven Vandenberghe. 2004. Convex Optimization. Cambridge University Press.
Liu, Nelson F., Kevin Lin, John Hewitt, et al. 2024. “Lost in the Middle: How Language Models Use Long Contexts.” Transactions of the Association for Computational Linguistics 12: 157–73.

Fan, Y. (2026). Spiritual Quotient Quantification, Prior-Conditioned Generation and Retrieval-Augmented Cognitive Agents: Multidimensional State Vectors, Personalized Prompt Conditioning and Cross-Corpus Synthesis [五维灵商量化架构、先验条件化生成与检索增强认知体] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090203 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).