What does RAG and Multi-Agent Reliability conclude, in numbers, about “High-Dimensional Geometry: Concentration of Measure, Effective Dimension and Hub Chunks”?

Answer: Chapter 1 uses concentration of measure, effective dimension and hubness to show that the larger the corpus, the higher the top-1 error (from 0.0730 to 0.5030 at 256 dimensions); that with 768 nominal dimensions and a power-law spectrum only 31.7 effective dimensions remain, putting the wrong chunk first among a hundred thousand with probability 0.9875; and that in a hundred dimensions 1 % of hub chunks intrude on the top ten of 77.7 % of unrelated questions.

Scope of the numbers: All data are synthetic, generated by the book's programs; the numbers show mechanisms and orders of magnitude and are not measurements of any product.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.0730, 0.5030, 256, 768

A. cosine of random unit vectors
d sd (simulated)1/sqrt(d) largest |cos| of 2,000
160.2524 0.2500 0.8060
640.1243 0.1250 0.4137
2560.0623 0.0625 0.1969
10240.0320 0.0312 0.1029
B. P(top-1 wrong), relevant cosine 0.30 (+ noise of same sd)
d M1000 M = 10000 M = 100000 M = 1000000
640.7890 0.9270 0.9710 0.9850
2560.0730 0.1740 0.3280 0.5030
7680.0000 0.0000 0.0000 0.0010
d64: sqrt(2 ln M / d) at M = 1e6 is 0.657
d768: sqrt(2 ln M / d) at M = 1e6 is 0.190
Reading: in this isotropic model more dimensions help - noise
cosines shrink like 1/sqrt(d) - but every tenfold growth of the
corpus raises the best distractor by a fixed step, so top-1
errors climb with M at any d. The next program shows
why real embeddings behave as if d were far smaller.

Key formula

\[d_{\mathrm{eff}} = \frac{\big(\sum_i \lambda_i\big)^2}{\sum_i \lambda_i^2} ,\]
Show the full program
# Chapter 1. p01: Concentration of measure and the corpus-size penalty.
# A. Cosines between independent random unit vectors in d dimensions
#    have mean 0 and standard deviation 1/sqrt(d) (2,000 pairs each).
# B. Retrieval as an extreme-value race. The relevant chunk has cosine
#    s = 0.30 with the query plus noise; each of M irrelevant chunks has a
#    cosine that is pure noise of sd 1/sqrt(d) (isotropic model). Top-1 is
#    wrong when the largest of M noise cosines beats the relevant one.
#    The largest of M grows like sqrt(2 ln M / d). 1,000 queries per cell.
import math
import random

def unit(rng, d):
    v = [rng.gauss(0, 1) for _ in range(d)]
    n = math.sqrt(sum(x * x for x in v))
    return [x / n for x in v]

def erfinv(y):
    # Newton on erf(x) = y, from a good start
    x = 0.0 if abs(y) < 0.7 else math.copysign(
        math.sqrt(-math.log((1 - abs(y)) / 2)), y)
    for _ in range(60):
        err = math.erf(x) - y
        x -= err / (2 / math.sqrt(math.pi) * math.exp(-x * x))
    return x

if __name__ == "__main__":
    rng = random.Random(61001)
    print("A. cosine of random unit vectors")
    print("      d    sd (simulated)   1/sqrt(d)   largest |cos| of 2,000")
    for d in (16, 64, 256, 1024):
        cs = []
        for _ in range(2000):
            a, b = unit(rng, d), unit(rng, d)
            cs.append(sum(x * y for x, y in zip(a, b)))
        sd = math.sqrt(sum(c * c for c in cs) / len(cs))
        print("   {:4d}      {:.4f}          {:.4f}       {:.4f}".format(
            d, sd, 1 / math.sqrt(d), max(abs(c) for c in cs)))
    print("\nB. P(top-1 wrong), relevant cosine 0.30 (+ noise of same sd)")
    sizes = (1000, 10000, 100000, 1000000)
    print("      d   " + "".join("   M = {:<8d}".format(m) for m in sizes))
    for d in (64, 256, 768):
        sd = 1 / math.sqrt(d)
        cells = []
        for m in sizes:
            wrong = 0
            for _ in range(1000):
                rel = 0.30 + rng.gauss(0, sd)
                # max of m iid N(0, sd^2) by inverse CDF of the max
                u = rng.random() ** (1 / m)
                z = math.sqrt(2) * erfinv(2 * u - 1)
                wrong += z * sd > rel
            cells.append("   {:9.4f}  ".format(wrong / 1000))
        print("   {:4d}".format(d) + "".join(cells))
    for d in (64, 768):
        print("   d = {:3d}: sqrt(2 ln M / d) at M = 1e6 is {:.3f}".format(
            d, math.sqrt(2 * math.log(1e6) / d)))
    print("\nReading: in this isotropic model more dimensions help - noise")
    print("cosines shrink like 1/sqrt(d) - but every tenfold growth of the")
    print("corpus raises the best distractor by a fixed step, so top-1")
    print("errors climb with M at any d. The next program shows")
    print("why real embeddings behave as if d were far smaller.")

Single source and citation

Fan, Y. (2026). Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades (Version 1.0.0), Chapter 1: High-Dimensional Geometry: Concentration of Measure, Effective Dimension and Hub Chunks. 5D Library. https://doi.org/10.5281/zenodo.24090170

DOI: 10.5281/zenodo.24090170 · Read the monograph · 5DL-B-061 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026061,
  author    = {Fan, Yafu},
  title     = {Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.24090170},
  url       = {https://doi.org/10.5281/zenodo.24090170}
}

Other chapter answers in this book