What does Citation Weight in Generative Search conclude, in numbers, about “Closing the Loop: an A/B Test of Answer-Paragraph Templates, the Citability Scorer and a Pre-Registered Test”?
Answer: Chapter 8 gives an A/B test of answer-paragraph templates, the Answer-Paragraph Citability Scorer and a pre-registered protocol (524 checks per group, simulated power 0.8085). Marked boxes separate published evidence, runnable Python calculations (21 programs), classical sources and the author's hypotheses.
Scope of the numbers: Its parameters are illustrative; all calculations are generated by its programs. It is not individual business advice.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.8085, 524
| p1 n / arm icc 0.05 icc 0.20 | |
| 0.14 | 1907 2194 3052 |
| 0.17 | 524 603 839 |
| 0.20 | 253 291 405 |
| simulated power at n | 524 per arm: 0.8085 (2000 runs) |
| icc 0.00 | 524 checks per arm = 4.4 weeks at 120 a week |
| icc 0.05 | 603 checks per arm = 5.0 weeks at 120 a week |
| icc 0.20 | 839 checks per arm = 7.0 weeks at 120 a week |
Show the full program
# Chapter 8. p82: how many checks the pre-registered test needs.
# Two-proportion test, alpha 0.05 two-sided, power 0.80, baseline
# citation rate p0 = 0.11 (close to arm A in p81). Checks are
# clustered: each question goes to m = 4 engines, so the needed count
# grows by the design effect 1 + (m - 1) icc. We then check the
# unclustered answer for p1 = 0.17 by 2000 seeded simulations.
# Rates and icc values are illustrative.
import math
import random
ZA, ZB = 1.959964, 0.841621
P0, M = 0.11, 4
def n_per_arm(p0, p1):
pbar = (p0 + p1) / 2
a = ZA * math.sqrt(2 * pbar * (1 - pbar))
b = ZB * math.sqrt(p0 * (1 - p0) + p1 * (1 - p1))
return math.ceil((a + b) ** 2 / (p1 - p0) ** 2)
print("%6s %10s %12s %12s" % ("p1", "n / arm", "icc 0.05", "icc 0.20"))
for p1 in (0.14, 0.17, 0.20):
n = n_per_arm(P0, p1)
d1, d2 = 1 + (M - 1) * 0.05, 1 + (M - 1) * 0.20
print("%6.2f %10d %12d %12d" % (p1, n, math.ceil(n * d1),
math.ceil(n * d2)))
rng = random.Random(20261113)
n = n_per_arm(P0, 0.17)
hits = 0
SIMS = 2000
for _ in range(SIMS):
ka = len([1 for _ in range(n) if rng.random() < P0])
kb = len([1 for _ in range(n) if rng.random() < 0.17])
pp = (ka + kb) / (2 * n)
se = math.sqrt(pp * (1 - pp) * 2 / n)
if se > 0 and (kb - ka) / n / se > ZA:
hits += 1
print("simulated power at n = %d per arm: %.4f (%d runs)" % (
n, hits / SIMS, SIMS))
for icc in (0.0, 0.05, 0.20):
need = math.ceil(n * (1 + (M - 1) * icc))
print("icc %.2f: %d checks per arm = %.1f weeks at 120 a week" % (
icc, need, need / 120))Single source and citation
Fan, Y. (2026). The Geometry of Citation Weight in Generative Search: Retrieval and Reranking Features and the Paragraph Topology of Highly Cited Bilingual Answer Pages (Version 1.0.0), Chapter 8: Closing the Loop: an A/B Test of Answer-Paragraph Templates, the Citability Scorer and a Pre-Registered Test. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-260 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026260,
author = {Fan, Yafu},
title = {The Geometry of Citation Weight in Generative Search: Retrieval and Reranking Features and the Paragraph Topology of Highly Cited Bilingual Answer Pages},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Measuring Visibility: Why Keyword Stuffing Fails and Why Sources, Figures and Quotations Help
- Chapter 2 · The Geometry of Hybrid Scoring: BM25 Saturation, Cosine Direction and Rank Fusion
- Chapter 3 · Paragraph Length and the Point of the Cut: Chunking, Truncation and an Interior Optimum
- Chapter 4 · The Discount by Citation Position: Cumulated Gain, Position-Adjusted Word Count and the Sag in the Middle
- Chapter 5 · Verifiability: Why Sources Raise the Chance of Citation, and the Trap of Inferring Features from Outside
- Chapter 6 · The Paragraph Topology of a Bilingual Direct-Answer Page: Self-Contained Paragraphs, One Set of Numbers on Four Surfaces, and Compliant Structured Data
- Chapter 7 · Classical Sources East and West: the Three Gauges of the Mozi and Garfield's Citation Index