What does The Nightly Funnel conclude, in numbers, about “Trustworthy Data: Negative Bias in Adaptive Data, Sample Ratio Mismatch and Guardrail Rollback”?

Answer: Chapter 6 the negative bias of adaptive data, sample ratio mismatch (a logging fault alone can manufacture a lift of 3.1%) and guardrail rollback.

Scope of the numbers: Its parameters are illustrative; all calculations are generated by its 23 programs; it is general business analysis, not business, legal or financial advice, and it promises no rise in conversion.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 3.1

critical chi-square 10.828; logged share of B 0.4924
logged visitorsfalse alarm (no fault)fault detected
2,0000.00100.004
10,0000.00180.037
50,0000.00060.559
200,0000.00081.000
null variant: true pay rate 0.050 on both arms; B measured 0.0515
phantom lift +3.1% from the fault alone
Show the full program
# Chapter 6. p62: sample ratio mismatch (SRM) check.
# Planned split 50/50. A logging fault silently drops 3 percent of B's
# visitors, and those dropped are visitors who left before the page
# finished loading, so none of them would have paid. Chi-square test
# with 1 degree of freedom at alpha = 0.001 (Kohavi et al. 2020 treat
# SRM as a trust check that blocks the read-out). Counts use a normal
# approximation to the binomial; 5,000 runs per size.
import math
import random
from statistics import NormalDist

rng = random.Random(40562)
CRIT = NormalDist().inv_cdf(1 - 0.001 / 2) ** 2
LOSS, REPS = 0.03, 5000
Q = 0.5 * (1 - LOSS) / (0.5 + 0.5 * (1 - LOSS))


def chi2(na, nb):
    e = (na + nb) / 2
    return (na - e) ** 2 / e + (nb - e) ** 2 / e


def rate(n, q):
    hits = 0
    for _ in range(REPS):
        nb = round(n * q + math.sqrt(n * q * (1 - q)) * rng.gauss(0, 1))
        hits += 1 if chi2(n - nb, nb) > CRIT else 0
    return hits / REPS


print("critical chi-square {:.3f}; logged share of B {:.4f}".format(
    CRIT, Q))
print("logged visitors   false alarm (no fault)   fault detected")
for n in [2000, 10000, 50000, 200000]:
    print("{:>15,} {:>24.4f} {:>16.3f}".format(n, rate(n, 0.5), rate(n, Q)))
p_true = 0.050
print("null variant: true pay rate {:.3f} on both arms; B measured "
      "{:.4f}".format(p_true, p_true / (1 - LOSS)))
print("  phantom lift {:+.1%} from the fault alone".format(
    1 / (1 - LOSS) - 1))

Single source and citation

Fan, Y. (2026). Autonomous Nightly Funnel Micro-Surgery, Bandit Traffic Allocation and Zero-Touch Conversion Self-Improvement (Version 1.0.0), Chapter 6: Trustworthy Data: Negative Bias in Adaptive Data, Sample Ratio Mismatch and Guardrail Rollback. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-405 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026405,
  author    = {Fan, Yafu},
  title     = {Autonomous Nightly Funnel Micro-Surgery, Bandit Traffic Allocation and Zero-Touch Conversion Self-Improvement},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book