What does Citation Weight in Generative Search conclude, in numbers, about “The Geometry of Hybrid Scoring: BM25 Saturation, Cosine Direction and Rank Fusion”?

Answer: Chapter 2 covers BM25 saturation, cosine direction and rank fusion: one new true matching term earns nearly three times what one repeated term does.

Scope of the numbers: Its parameters are illustrative; all calculations are generated by its programs. It is not individual business advice.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python only

k1 = 1.2, b = 0.75, avgdl = 60, idf = 2.0
extradlw(t1)w(t2)totalmarginal
0602.00002.00004.0000-
1612.73721.98654.7236+0.7236
2623.12061.97315.0937+0.3700
3633.35561.95995.3155+0.2218
5653.62891.93415.5629+0.1237
8683.83721.89665.7338+0.0569
12723.97771.84875.8265+0.0232
20804.10671.76005.8667+0.0050
401004.21501.57145.7864-0.0040
ceiling of w(t1) as tf grows: idf (k1 + 1) = 4.4000
one new matching term t3 : total 5.9594 (+1.9594)
one more repetition of t1: total 4.7236 (+0.7236)

Key formula

\[w = \mathrm{idf} \cdot \frac{\mathrm{tf}\,(k_1+1)}{\mathrm{tf} + k_1\left(1-b+b\,\frac{dl}{avgdl}\right)}.\]
Show the full program
# Chapter 2. p21: BM25 saturation, the arithmetic behind stuffing.
# Okapi BM25 term weight (Robertson and Zaragoza 2009):
#   w = idf * tf (k1 + 1) / (tf + k1 (1 - b + b dl / avgdl))
# A 60-token passage answers a two-term query (t1, t2) once each. We
# repeat t1 r more times (each repetition adds one token) and compare
# with adding one new matching term t3 of the same idf instead.
# k1 = 1.2, b = 0.75 are common defaults; idf = 2.0 is illustrative.
import math

K1, B, AVGDL, IDF = 1.2, 0.75, 60.0, 2.0


def w(tf, dl):
    if tf == 0:
        return 0.0
    return IDF * tf * (K1 + 1) / (tf + K1 * (1 - B + B * dl / AVGDL))


print("k1 = %.1f, b = %.2f, avgdl = %.0f, idf = %.1f" % (K1, B, AVGDL,
                                                        IDF))
print("%6s %4s %8s %8s %8s %9s" % ("extra", "dl", "w(t1)", "w(t2)",
                                   "total", "marginal"))
prev = None
for r in (0, 1, 2, 3, 5, 8, 12, 20, 40):
    dl = 60 + r
    total = math.fsum([w(1 + r, dl), w(1, dl)])
    mg = "-" if prev is None else "%+.4f" % (total - prev[1])
    if prev is not None and r - prev[0] > 1:
        mg = "%+.4f" % ((total - prev[1]) / (r - prev[0]))
    print("%6d %4d %8.4f %8.4f %8.4f %9s" % (r, dl, w(1 + r, dl),
                                             w(1, dl), total, mg))
    prev = (r, total)
cap = IDF * (K1 + 1)
print("ceiling of w(t1) as tf grows: idf (k1 + 1) = %.4f" % cap)
base = w(1, 60) * 2
new = w(1, 61) * 3
rep = math.fsum([w(2, 61), w(1, 61)])
print("one new matching term t3 : total %.4f (+%.4f)" % (new,
                                                        new - base))
print("one more repetition of t1: total %.4f (+%.4f)" % (rep,
                                                        rep - base))

Single source and citation

Fan, Y. (2026). The Geometry of Citation Weight in Generative Search: Retrieval and Reranking Features and the Paragraph Topology of Highly Cited Bilingual Answer Pages (Version 1.0.0), Chapter 2: The Geometry of Hybrid Scoring: BM25 Saturation, Cosine Direction and Rank Fusion. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-260 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026260,
  author    = {Fan, Yafu},
  title     = {The Geometry of Citation Weight in Generative Search: Retrieval and Reranking Features and the Paragraph Topology of Highly Cited Bilingual Answer Pages},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book