What does RAG and Multi-Agent Reliability conclude, in numbers, about “Lexical and Dense Retrieval: Their Blind Spots and the Smoothing Constant of Reciprocal Rank Fusion”?

Answer: Chapter 2 shows that dense retrieval cannot recognise identifiers and lexical retrieval cannot recognise paraphrases; fusing complete rankings leaves both query types at a recall@1 of 0.055, and fusing short lists leaves identifier queries at 0.475; reciprocal rank fusion's smoothing constant k = 60 is not the best for recall@1 in any of three scenarios, and with one strong and one weak retriever no k beats the strong one alone (0.604).

Scope of the numbers: All data are synthetic, generated by the book's programs; the numbers show mechanisms and orders of magnitude and are not measurements of any product.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.055, 0.475

   query type     ranker   recall@1   recall@10
   paraphrase     BM25       0.005      0.005
   paraphrase     dense      0.980      1.000
   paraphrase     RRF-all    0.055      0.740
   paraphrase     RRF-100    0.980      1.000
   exact code     BM25       1.000      1.000
   exact code     dense      0.000      0.005
   exact code     RRF-all    0.055      0.750
   exact code     RRF-100    0.475      1.000
Reading: BM25 cannot find a chunk that shares no words with
the query; the dense model cannot tell one code from another,
because every code is nearly the same vector. Fusion is not
automatically the best of both: fusing full rankings lets the
blind ranker's noise outvote the one that knows. Fusing short
lists, with zero-score chunks left out, repairs paraphrase
queries completely but leaves exact-code queries a near coin
toss: the dense top hit ties with the true chunk.

Key formula

\[\mathrm{RRF}(d) = \sum_{i} \frac{1}{k + r_i(d)} ,\]
Show the full program
# Chapter 2. p04: Lexical (BM25) and dense retrieval fail on different
# queries.
# Synthetic corpus: 2,000 chunks in 50 topics. Each chunk has 20 topic
# words, 3 detail words of its own, and a unique error code such as
# E1043. Two query types, 200 each:
#   paraphrase  3 topic synonyms + 3 paraphrased detail words - no word
#               in common with the gold chunk
#   exact code  "what does code E1043 mean" - the code is the only clue
# BM25 (k1 = 1.2, b = 0.75) scores exact token overlap. The dense model
# (d = 64) maps a word and its synonym/paraphrase to nearby vectors, but
# like a subword tokenizer sees every code as almost the same token: a
# shared "code" vector plus only 0.15 of a code-specific part. A chunk
# or query vector is the normalised mean of its word vectors.
# Hybrid: reciprocal rank fusion, score = sum 1 / (60 + rank), done two
# ways: RRF-all fuses complete rankings (ties broken at random); RRF-100
# fuses each ranker's top-100 list only, and BM25 lists only chunks with
# a positive score - the usual implementation.
import math
import random

D, TOPICS, DOCS = 64, 50, 2000

def unit(rng):
    v = [rng.gauss(0, 1) for _ in range(D)]
    n = math.sqrt(sum(x * x for x in v))
    return [x / n for x in v]

def mix(*parts):
    v = [0.0] * D
    for w, vec in parts:
        v = [a + w * b for a, b in zip(v, vec)]
    return v

def norm(v):
    n = math.sqrt(sum(x * x for x in v)) or 1.0
    return [x / n for x in v]

class Corpus:
    def __init__(self, rng):
        self.rng = rng
        self.topic = [unit(rng) for _ in range(TOPICS)]
        self.code_dir = unit(rng)
        self.vec = {}
        self.docs = []
        for i in range(DOCS):
            t = i % TOPICS
            words = ["t{}w{}".format(t, rng.randrange(40)) for _ in range(20)]
            words += ["d{}x{}".format(i, k) for k in range(3)]
            words += ["E{}".format(1000 + i)]
            self.docs.append((t, words))
        self.df = {}
        for _, ws in self.docs:
            for w in set(ws):
                self.df[w] = self.df.get(w, 0) + 1
        self.avg = sum(len(ws) for _, ws in self.docs) / DOCS
        self.emb = [self.embed(ws) for _, ws in self.docs]

    def wvec(self, w):
        if w in self.vec:
            return self.vec[w]
        r = self.rng
        if w.startswith("t"):                       # topic word or synonym
            t = int(w[1:].replace("s", "w").split("w")[0])
            v = mix((1.0, self.topic[t]), (0.5, unit(r)))
        elif w.startswith("d"):                     # detail word/paraphrase
            base = w.replace("p", "x")
            if base not in self.vec:
                self.vec[base] = unit(r)
            v = mix((1.0, self.vec[base]), (0.3, unit(r)))
            if w == base:
                return self.vec[base]
        elif w.startswith("E"):                     # code: almost one token
            v = mix((1.0, self.code_dir), (0.15, unit(r)))
        else:
            v = unit(r)
        self.vec[w] = v
        return v

    def embed(self, words):
        return norm(mix(*[(1.0, self.wvec(w)) for w in words]))

    def bm25(self, q):
        out = []
        for i, (_, ws) in enumerate(self.docs):
            s = 0.0
            for w in set(q):
                f = ws.count(w)
                if f:
                    n = self.df[w]
                    idf = math.log(1 + (DOCS - n + 0.5) / (n + 0.5))
                    s += idf * f * 2.2 / (f + 1.2 * (0.25 + 0.75 * len(ws)
                                                     / self.avg))
            out.append(s)
        return out

    def dense(self, q):
        qv = self.embed(q)
        return [sum(a * b for a, b in zip(qv, e)) for e in self.emb]

def ranks(scores, rng):
    order = sorted(range(len(scores)),
                   key=lambda i: (-scores[i], rng.random()))
    r = [0] * len(scores)
    for pos, i in enumerate(order):
        r[i] = pos + 1
    return r

if __name__ == "__main__":
    rng = random.Random(61004)
    c = Corpus(rng)
    res = {}
    for qtype in ("paraphrase", "exact code"):
        hits = {"BM25": [0, 0], "dense": [0, 0], "RRF-all": [0, 0],
                "RRF-100": [0, 0]}
        for _ in range(200):
            g = rng.randrange(DOCS)
            t = c.docs[g][0]
            if qtype == "paraphrase":
                q = ["t{}s{}".format(t, rng.randrange(40)) for _ in range(3)]
                q += ["d{}p{}".format(g, k) for k in range(3)]
            else:
                q = ["what", "does", "code", "E{}".format(1000 + g), "mean"]
            sb = c.bm25(q)
            rb = ranks(sb, rng)
            rd = ranks(c.dense(q), rng)
            fused = [1 / (60 + a) + 1 / (60 + b) for a, b in zip(rb, rd)]
            rf = ranks(fused, rng)
            f100 = [(1 / (60 + a) if a <= 100 and s > 0 else 0.0)
                    + (1 / (60 + b) if b <= 100 else 0.0)
                    for a, b, s in zip(rb, rd, sb)]
            r100 = ranks(f100, rng)
            for name, r in (("BM25", rb), ("dense", rd), ("RRF-all", rf),
                            ("RRF-100", r100)):
                hits[name][0] += r[g] == 1
                hits[name][1] += r[g] <= 10
        res[qtype] = hits
    print("   query type     ranker   recall@1   recall@10")
    for qtype, hits in res.items():
        for name, (h1, h10) in hits.items():
            print("   {:<13}  {:<7}  {:7.3f}    {:7.3f}".format(
                qtype, name, h1 / 200, h10 / 200))
    print("\nReading: BM25 cannot find a chunk that shares no words with")
    print("the query; the dense model cannot tell one code from another,")
    print("because every code is nearly the same vector. Fusion is not")
    print("automatically the best of both: fusing full rankings lets the")
    print("blind ranker's noise outvote the one that knows. Fusing short")
    print("lists, with zero-score chunks left out, repairs paraphrase")
    print("queries completely but leaves exact-code queries a near coin")
    print("toss: the dense top hit ties with the true chunk.")

Single source and citation

Fan, Y. (2026). Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades (Version 1.0.0), Chapter 2: Lexical and Dense Retrieval: Their Blind Spots and the Smoothing Constant of Reciprocal Rank Fusion. 5D Library. https://doi.org/10.5281/zenodo.24090170

DOI: 10.5281/zenodo.24090170 · Read the monograph · 5DL-B-061 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026061,
  author    = {Fan, Yafu},
  title     = {Retrieval-Augmented Generation Geometry, Reciprocal Rank Fusion and Multi-Agent Error Cascades},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.24090170},
  url       = {https://doi.org/10.5281/zenodo.24090170}
}

Other chapter answers in this book