What does RAG and Multi-Agent Reliability conclude, in numbers, about “Lexical and Dense Retrieval: Their Blind Spots and the Smoothing Constant of Reciprocal Rank Fusion”?
Answer: Chapter 2 shows that dense retrieval cannot recognise identifiers and lexical retrieval cannot recognise paraphrases; fusing complete rankings leaves both query types at a recall@1 of 0.055, and fusing short lists leaves identifier queries at 0.475; reciprocal rank fusion's smoothing constant k = 60 is not the best for recall@1 in any of three scenarios, and with one strong and one weak retriever no k beats the strong one alone (0.604).
Scope of the numbers: All data are synthetic, generated by the book's programs; the numbers show mechanisms and orders of magnitude and are not measurements of any product.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.055, 0.475
query type ranker recall@1 recall@10 paraphrase BM25 0.005 0.005 paraphrase dense 0.980 1.000 paraphrase RRF-all 0.055 0.740 paraphrase RRF-100 0.980 1.000 exact code BM25 1.000 1.000 exact code dense 0.000 0.005 exact code RRF-all 0.055 0.750 exact code RRF-100 0.475 1.000 Reading: BM25 cannot find a chunk that shares no words with the query; the dense model cannot tell one code from another, because every code is nearly the same vector. Fusion is not automatically the best of both: fusing full rankings lets the blind ranker's noise outvote the one that knows. Fusing short lists, with zero-score chunks left out, repairs paraphrase queries completely but leaves exact-code queries a near coin toss: the dense top hit ties with the true chunk.
Key formula
Show the full program
# Chapter 2. p04: Lexical (BM25) and dense retrieval fail on different
# queries.
# Synthetic corpus: 2,000 chunks in 50 topics. Each chunk has 20 topic
# words, 3 detail words of its own, and a unique error code such as
# E1043. Two query types, 200 each:
# paraphrase 3 topic synonyms + 3 paraphrased detail words - no word
# in common with the gold chunk
# exact code "what does code E1043 mean" - the code is the only clue
# BM25 (k1 = 1.2, b = 0.75) scores exact token overlap. The dense model
# (d = 64) maps a word and its synonym/paraphrase to nearby vectors, but
# like a subword tokenizer sees every code as almost the same token: a
# shared "code" vector plus only 0.15 of a code-specific part. A chunk
# or query vector is the normalised mean of its word vectors.
# Hybrid: reciprocal rank fusion, score = sum 1 / (60 + rank), done two
# ways: RRF-all fuses complete rankings (ties broken at random); RRF-100
# fuses each ranker's top-100 list only, and BM25 lists only chunks with
# a positive score - the usual implementation.
import math
import random
D, TOPICS, DOCS = 64, 50, 2000
def unit(rng):
v = [rng.gauss(0, 1) for _ in range(D)]
n = math.sqrt(sum(x * x for x in v))
return [x / n for x in v]
def mix(*parts):
v = [0.0] * D
for w, vec in parts:
v = [a + w * b for a, b in zip(v, vec)]
return v
def norm(v):
n = math.sqrt(sum(x * x for x in v)) or 1.0
return [x / n for x in v]
class Corpus:
def __init__(self, rng):
self.rng = rng
self.topic = [unit(rng) for _ in range(TOPICS)]
self.code_dir = unit(rng)
self.vec = {}
self.docs = []
for i in range(DOCS):
t = i % TOPICS
words = ["t{}w{}".format(t, rng.randrange(40)) for _ in range(20)]
words += ["d{}x{}".format(i, k) for k in range(3)]
words += ["E{}".format(1000 + i)]
self.docs.append((t, words))
self.df = {}
for _, ws in self.docs:
for w in set(ws):
self.df[w] = self.df.get(w, 0) + 1
self.avg = sum(len(ws) for _, ws in self.docs) / DOCS
self.emb = [self.embed(ws) for _, ws in self.docs]
def wvec(self, w):
if w in self.vec:
return self.vec[w]
r = self.rng
if w.startswith("t"): # topic word or synonym
t = int(w[1:].replace("s", "w").split("w")[0])
v = mix((1.0, self.topic[t]), (0.5, unit(r)))
elif w.startswith("d"): # detail word/paraphrase
base = w.replace("p", "x")
if base not in self.vec:
self.vec[base] = unit(r)
v = mix((1.0, self.vec[base]), (0.3, unit(r)))
if w == base:
return self.vec[base]
elif w.startswith("E"): # code: almost one token
v = mix((1.0, self.code_dir), (0.15, unit(r)))
else:
v = unit(r)
self.vec[w] = v
return v
def embed(self, words):
return norm(mix(*[(1.0, self.wvec(w)) for w in words]))
def bm25(self, q):
out = []
for i, (_, ws) in enumerate(self.docs):
s = 0.0
for w in set(q):
f = ws.count(w)
if f:
n = self.df[w]
idf = math.log(1 + (DOCS - n + 0.5) / (n + 0.5))
s += idf * f * 2.2 / (f + 1.2 * (0.25 + 0.75 * len(ws)
/ self.avg))
out.append(s)
return out
def dense(self, q):
qv = self.embed(q)
return [sum(a * b for a, b in zip(qv, e)) for e in self.emb]
def ranks(scores, rng):
order = sorted(range(len(scores)),
key=lambda i: (-scores[i], rng.random()))
r = [0] * len(scores)
for pos, i in enumerate(order):
r[i] = pos + 1
return r
if __name__ == "__main__":
rng = random.Random(61004)
c = Corpus(rng)
res = {}
for qtype in ("paraphrase", "exact code"):
hits = {"BM25": [0, 0], "dense": [0, 0], "RRF-all": [0, 0],
"RRF-100": [0, 0]}
for _ in range(200):
g = rng.randrange(DOCS)
t = c.docs[g][0]
if qtype == "paraphrase":
q = ["t{}s{}".format(t, rng.randrange(40)) for _ in range(3)]
q += ["d{}p{}".format(g, k) for k in range(3)]
else:
q = ["what", "does", "code", "E{}".format(1000 + g), "mean"]
sb = c.bm25(q)
rb = ranks(sb, rng)
rd = ranks(c.dense(q), rng)
fused = [1 / (60 + a) + 1 / (60 + b) for a, b in zip(rb, rd)]
rf = ranks(fused, rng)
f100 = [(1 / (60 + a) if a <= 100 and s > 0 else 0.0)
+ (1 / (60 + b) if b <= 100 else 0.0)
for a, b, s in zip(rb, rd, sb)]
r100 = ranks(f100, rng)
for name, r in (("BM25", rb), ("dense", rd), ("RRF-all", rf),
("RRF-100", r100)):
hits[name][0] += r[g] == 1
hits[name][1] += r[g] <= 10
res[qtype] = hits
print(" query type ranker recall@1 recall@10")
for qtype, hits in res.items():
for name, (h1, h10) in hits.items():
print(" {:<13} {:<7} {:7.3f} {:7.3f}".format(
qtype, name, h1 / 200, h10 / 200))
print("\nReading: BM25 cannot find a chunk that shares no words with")
print("the query; the dense model cannot tell one code from another,")
print("because every code is nearly the same vector. Fusion is not")
print("automatically the best of both: fusing full rankings lets the")
print("blind ranker's noise outvote the one that knows. Fusing short")
print("lists, with zero-score chunks left out, repairs paraphrase")
print("queries completely but leaves exact-code queries a near coin")
print("toss: the dense top hit ties with the true chunk.")Single source and citation
Fan, Y. (2026). Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades (Version 1.0.0), Chapter 2: Lexical and Dense Retrieval: Their Blind Spots and the Smoothing Constant of Reciprocal Rank Fusion. 5D Library. https://doi.org/10.5281/zenodo.24090170
DOI: 10.5281/zenodo.24090170 · Read the monograph · 5DL-B-061 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026061,
author = {Fan, Yafu},
title = {Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.24090170},
url = {https://doi.org/10.5281/zenodo.24090170}
}Other chapter answers in this book
- Chapter 1 · High-Dimensional Geometry: Concentration of Measure, Effective Dimension and Hub Chunks
- Chapter 3 · The Context Window: Position Decay, Evidence Dilution and the Best Number of Chunks
- Chapter 4 · Multi-Agent Pipelines: Error Cascades and the Verifier-Gated Absorbing Markov Chain
- Chapter 5 · Verifier Independence and the Cost of Retries
- Chapter 6 · The Statistics of Evaluation Itself: Sample Size, Judge Bias and the Winner's Curse
- Chapter 7 · The Routing of the *Qilue* and Han Fei's *Canyan*: Two Ancient Methods of Retrieval and Verification
- Chapter 8 · A Five-Step Check and a Pre-Registered Offline Evaluation