What does Citation Weight in Generative Search conclude, in numbers, about “Closing the Loop: an A/B Test of Answer-Paragraph Templates, the Citability Scorer and a Pre-Registered Test”?

Answer: Chapter 8 gives an A/B test of answer-paragraph templates, the Answer-Paragraph Citability Scorer and a pre-registered protocol (524 checks per group, simulated power 0.8085). Marked boxes separate published evidence, runnable Python calculations (21 programs), classical sources and the author's hypotheses.

Scope of the numbers: Its parameters are illustrative; all calculations are generated by its programs. It is not individual business advice.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.8085, 524

p1 n / arm icc 0.05 icc 0.20
0.141907 2194 3052
0.17524 603 839
0.20253 291 405
simulated power at n524 per arm: 0.8085 (2000 runs)
icc 0.00524 checks per arm = 4.4 weeks at 120 a week
icc 0.05603 checks per arm = 5.0 weeks at 120 a week
icc 0.20839 checks per arm = 7.0 weeks at 120 a week
Show the full program
# Chapter 8. p82: how many checks the pre-registered test needs.
# Two-proportion test, alpha 0.05 two-sided, power 0.80, baseline
# citation rate p0 = 0.11 (close to arm A in p81). Checks are
# clustered: each question goes to m = 4 engines, so the needed count
# grows by the design effect 1 + (m - 1) icc. We then check the
# unclustered answer for p1 = 0.17 by 2000 seeded simulations.
# Rates and icc values are illustrative.
import math
import random

ZA, ZB = 1.959964, 0.841621
P0, M = 0.11, 4


def n_per_arm(p0, p1):
    pbar = (p0 + p1) / 2
    a = ZA * math.sqrt(2 * pbar * (1 - pbar))
    b = ZB * math.sqrt(p0 * (1 - p0) + p1 * (1 - p1))
    return math.ceil((a + b) ** 2 / (p1 - p0) ** 2)


print("%6s %10s %12s %12s" % ("p1", "n / arm", "icc 0.05", "icc 0.20"))
for p1 in (0.14, 0.17, 0.20):
    n = n_per_arm(P0, p1)
    d1, d2 = 1 + (M - 1) * 0.05, 1 + (M - 1) * 0.20
    print("%6.2f %10d %12d %12d" % (p1, n, math.ceil(n * d1),
                                    math.ceil(n * d2)))
rng = random.Random(20261113)
n = n_per_arm(P0, 0.17)
hits = 0
SIMS = 2000
for _ in range(SIMS):
    ka = len([1 for _ in range(n) if rng.random() < P0])
    kb = len([1 for _ in range(n) if rng.random() < 0.17])
    pp = (ka + kb) / (2 * n)
    se = math.sqrt(pp * (1 - pp) * 2 / n)
    if se > 0 and (kb - ka) / n / se > ZA:
        hits += 1
print("simulated power at n = %d per arm: %.4f (%d runs)" % (
    n, hits / SIMS, SIMS))
for icc in (0.0, 0.05, 0.20):
    need = math.ceil(n * (1 + (M - 1) * icc))
    print("icc %.2f: %d checks per arm = %.1f weeks at 120 a week" % (
        icc, need, need / 120))

Single source and citation

Fan, Y. (2026). The Geometry of Citation Weight in Generative Search: Retrieval and Reranking Features and the Paragraph Topology of Highly Cited Bilingual Answer Pages (Version 1.0.0), Chapter 8: Closing the Loop: an A/B Test of Answer-Paragraph Templates, the Citability Scorer and a Pre-Registered Test. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-260 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026260,
  author    = {Fan, Yafu},
  title     = {The Geometry of Citation Weight in Generative Search: Retrieval and Reranking Features and the Paragraph Topology of Highly Cited Bilingual Answer Pages},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book