5D Library › Spiritual Quotient Quantification, Prior-Conditioned Generation and Retrieval-Augmented Cognitive Agents · 中文

Chapter 5. Retrieval Augmentation: One-Book Recall and Cross-Book Synthesis

Retrieval-augmented generation

Retrieval-augmented generation (RAG) joins a retriever to a generative model: first find the passages in a document collection most relevant to a question, then let the model answer from them (Lewis et al. 2020). For questions about a single fact the method works well. The author’s design goes further: let the model draw on several of the library’s 82 volumes at once and synthesise across them. This chapter simulates both kinds of question.

A simulated library

The program takes 82 volumes (the whole library) of 40 passages each, embedded in a 24-dimensional space: each volume has a topic direction, and its passages mix that direction with directions of their own. A one-book question points at one passage plus noise of size σ\sigma; a cross-book question points at the mean of one passage from each of three volumes, plus noise. Passages are ranked by cosine similarity.

Calculation

# Chapter 5. p09: Recall at k in an 82-volume library, one book or three.
# Each of 82 volumes has 40 passages. In a 24-dimensional embedding, a
# volume has a topic vector t, and its passages are 0.8 t + 0.6 u (u its
# own direction), normalised. A single-book question points at one
# passage plus noise of size sigma; a cross-book question points at the
# mean of three passages from three different volumes plus noise.
# Passages are ranked by cosine similarity. Over 150 questions of each
# kind (fixed seed) the program reports recall at k: the share of
# questions whose needed passage (or all three) are in the top k.
import math
import random

BOOKS, PER, DIM = 82, 40, 24

def unit(v):
    n = math.sqrt(sum(x * x for x in v))
    return [x / n for x in v]

def corpus(rng):
    out = []
    for b in range(BOOKS):
        t = unit([rng.gauss(0, 1) for _ in range(DIM)])
        for _ in range(PER):
            u = unit([rng.gauss(0, 1) for _ in range(DIM)])
            out.append((b, unit([0.8 * x + 0.6 * y for x, y in zip(t, u)])))
    return out

def ranks(cor, q):
    q = unit(q)
    sc = [sum(a * b for a, b in zip(v, q)) for _, v in cor]
    return sorted(range(len(cor)), key=lambda i: -sc[i])

def run(rng, cor, sigma, nneed, ks):
    hits = {k: 0 for k in ks}
    for _ in range(150):
        books = rng.sample(range(BOOKS), nneed)
        need = [b * PER + rng.randrange(PER) for b in books]
        q = [sum(cor[i][1][d] for i in need) / nneed +
             sigma * rng.gauss(0, 1) / math.sqrt(DIM) for d in range(DIM)]
        order = ranks(cor, q)
        pos = {i: r for r, i in enumerate(order)}
        worst = max(pos[i] for i in need)
        for k in ks:
            hits[k] += worst < k
    return {k: hits[k] / 150 for k in ks}

if __name__ == "__main__":
    rng = random.Random(76009)
    cor = corpus(rng)
    ks = (5, 10, 20, 40)
    print("   question   sigma  " + " ".join("{:>7s}".format(
        "k={}".format(k)) for k in ks))
    for nneed, name in ((1, "one book"), (3, "three")):
        for sigma in (0.5, 1.0):
            r = run(rng, cor, sigma, nneed, ks)
            print("   {:9s} {:6.1f}  ".format(name, sigma) + " ".join(
                "{:7.3f}".format(r[k]) for k in ks))
    print("\nReading: a question about one book finds its passage in a")
    print("short list. A question that needs three books at once is far")
    print("harder: all three must outrank thousands of passages, and")
    print("plain top-k retrieval misses them even at k = 40.")
#    question   sigma      k=5    k=10    k=20    k=40
#    one book     0.5    1.000   1.000   1.000   1.000
#    one book     1.0    0.913   0.980   0.987   0.993
#    three        0.5    0.000   0.007   0.013   0.060
#    three        1.0    0.000   0.000   0.000   0.000
#
# Reading: a question about one book finds its passage in a
# short list. A question that needs three books at once is far
# harder: all three must outrank thousands of passages, and
# plain top-k retrieval misses them even at k = 40.
Recall against the number of passages retrieved, for one-book and three-book questions.

A one-book question almost always finds its passage in a very short list: with noise 0.5 recall in the top 5 is 1.000; with noise 1.0 it is 0.913 in the top 5 and 0.993 in the top 40. Cross-book questions are another matter: requiring all three passages in the top kk, with noise 0.5 the top 20 succeed only 0.013 of the time and the top 40 only 0.060; with noise 1.0, never. A question vector blending three volumes is not close enough to any of their particular passages.

Quotas and splitting

A natural remedy is diversity: at most one passage per volume, forcing the results to cover more volumes (akin to maximal-marginal-relevance reranking) (Carbonell and Goldstein 1998). Another is to split the question into its three parts first and retrieve for each. The program compares three strategies; when splitting, the model splits correctly with probability ss and falls back to direct retrieval otherwise.

Calculation

# Chapter 5. p10: Finding three books at once: quotas and decomposition.
# Same library model as p09 (82 volumes x 40 passages, 24 dimensions)
# and cross-book questions needing one passage from each of three
# volumes, with noise sigma = 0.5. Three strategies with k passages:
#   PLAIN  - top k by cosine to the whole question;
#   QUOTA  - at most one passage per volume, best volumes first (so k
#            distinct volumes);
#   SPLIT  - the agent splits the question into its three parts and
#            retrieves top k/3 for each; it splits correctly with
#            probability s (1.0 or 0.7), otherwise it falls back to
#            PLAIN.
# Success: all three needed passages retrieved. 150 questions each.
import math
import random

BOOKS, PER, DIM = 82, 40, 24

def unit(v):
    n = math.sqrt(sum(x * x for x in v))
    return [x / n for x in v]

def corpus(rng):
    out = []
    for b in range(BOOKS):
        t = unit([rng.gauss(0, 1) for _ in range(DIM)])
        for _ in range(PER):
            u = unit([rng.gauss(0, 1) for _ in range(DIM)])
            out.append(unit([0.8 * x + 0.6 * y for x, y in zip(t, u)]))
    return out

def order(cor, q):
    q = unit(q)
    sc = [sum(a * b for a, b in zip(v, q)) for v in cor]
    return sorted(range(len(cor)), key=lambda i: -sc[i])

def noisy(rng, v, sigma=0.5):
    return [x + sigma * rng.gauss(0, 1) / math.sqrt(DIM) for x in v]

def trial(rng, cor, k, strategy, s=1.0):
    books = rng.sample(range(BOOKS), 3)
    need = [b * PER + rng.randrange(PER) for b in books]
    whole = noisy(rng, [sum(cor[i][d] for i in need) / 3
                        for d in range(DIM)])
    if strategy == "SPLIT" and rng.random() < s:
        got = set()
        for i in need:
            got |= set(order(cor, noisy(rng, cor[i]))[:k // 3])
    elif strategy == "QUOTA":
        got, seen = set(), set()
        for i in order(cor, whole):
            if i // PER not in seen:
                seen.add(i // PER)
                got.add(i)
                if len(got) == k:
                    break
    else:
        got = set(order(cor, whole)[:k])
    return all(i in got for i in need)

if __name__ == "__main__":
    rng = random.Random(76010)
    cor = corpus(rng)
    print("   strategy      k=15    k=30")
    for name, s in (("PLAIN", 1.0), ("QUOTA", 1.0), ("SPLIT", 1.0),
                    ("SPLIT", 0.7)):
        row = [sum(trial(rng, cor, k, name, s) for _ in range(150)) / 150
               for k in (15, 30)]
        label = name if s == 1.0 else "{} s={}".format(name, s)
        print("   {:12s} {:6.3f} {:7.3f}".format(label, *row))
    print("\nReading: diversity quotas do not rescue a blended question,")
    print("because the best passage of a needed volume is not the one")
    print("the question points at. Splitting the question into its parts")
    print("does: synthesis across books is a planning problem first and")
    print("a retrieval problem second.")
#    strategy      k=15    k=30
#    PLAIN         0.013   0.073
#    QUOTA         0.007   0.000
#    SPLIT         1.000   1.000
#    SPLIT s=0.7   0.687   0.767
#
# Reading: diversity quotas do not rescue a blended question,
# because the best passage of a needed volume is not the one
# the question points at. Splitting the question into its parts
# does: synthesis across books is a planning problem first and
# a retrieval problem second.

Diversity quotas barely help (0.007 in the top 15): the passage of each volume most similar to a blended question is not the one the question needs. Splitting the question changes everything: split correctly, success is 1.000; with only seven splits in ten correct, success lies between 0.687 and 0.767, set by the accuracy of the split. Synthesis across books is first a planning problem, deciding what to look for in which volumes, and only then a retrieval problem. The library graph of Chapter 6 is the map such planning can rely on.

Hypothesis · not established

The author’s interpretation (not yet tested): in the author’s question-answering system over this library, having the model split cross-book questions into per-volume sub-questions with the help of the library graph, then retrieve for each, clearly raises the accuracy of cross-book synthesis. What would show this wrong: if, on a pre-registered evaluation set, split retrieval and direct retrieval did not differ clearly in accuracy on cross-book questions, cross-book questions in real embeddings would not be as hard as in this chapter’s model.

Summary

One-book questions find their passage in a very short list; questions needing one passage from each of three volumes succeed only 0.060 of the time with direct retrieval even in the top 40. One-passage-per-volume quotas do not help; splitting the question first makes success depend on the accuracy of the split.

References

Carbonell, Jaime, and Jade Goldstein. 1998. “The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries.” Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 335–36.
Lewis, Patrick, Ethan Perez, Aleksandra Piktus, et al. 2020. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” Advances in Neural Information Processing Systems 33, 9459–74.

Fan, Y. (2026). Spiritual Quotient Quantification, Prior-Conditioned Generation and Retrieval-Augmented Cognitive Agents: Multidimensional State Vectors, Personalized Prompt Conditioning and Cross-Corpus Synthesis [五维灵商量化架构、先验条件化生成与检索增强认知体] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090203 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).