What does The Nightly Funnel conclude, in numbers, about “Trustworthy Data: Negative Bias in Adaptive Data, Sample Ratio Mismatch and Guardrail Rollback”?
Answer: Chapter 6 the negative bias of adaptive data, sample ratio mismatch (a logging fault alone can manufacture a lift of 3.1%) and guardrail rollback.
Scope of the numbers: Its parameters are illustrative; all calculations are generated by its 23 programs; it is general business analysis, not business, legal or financial advice, and it promises no rise in conversion.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 3.1
| critical chi-square 10.828; logged share of B 0.4924 | ||
| logged visitors | false alarm (no fault) | fault detected |
| 2,000 | 0.0010 | 0.004 |
| 10,000 | 0.0018 | 0.037 |
| 50,000 | 0.0006 | 0.559 |
| 200,000 | 0.0008 | 1.000 |
| null variant: true pay rate 0.050 on both arms; B measured 0.0515 | ||
| phantom lift +3.1% from the fault alone |
Show the full program
# Chapter 6. p62: sample ratio mismatch (SRM) check.
# Planned split 50/50. A logging fault silently drops 3 percent of B's
# visitors, and those dropped are visitors who left before the page
# finished loading, so none of them would have paid. Chi-square test
# with 1 degree of freedom at alpha = 0.001 (Kohavi et al. 2020 treat
# SRM as a trust check that blocks the read-out). Counts use a normal
# approximation to the binomial; 5,000 runs per size.
import math
import random
from statistics import NormalDist
rng = random.Random(40562)
CRIT = NormalDist().inv_cdf(1 - 0.001 / 2) ** 2
LOSS, REPS = 0.03, 5000
Q = 0.5 * (1 - LOSS) / (0.5 + 0.5 * (1 - LOSS))
def chi2(na, nb):
e = (na + nb) / 2
return (na - e) ** 2 / e + (nb - e) ** 2 / e
def rate(n, q):
hits = 0
for _ in range(REPS):
nb = round(n * q + math.sqrt(n * q * (1 - q)) * rng.gauss(0, 1))
hits += 1 if chi2(n - nb, nb) > CRIT else 0
return hits / REPS
print("critical chi-square {:.3f}; logged share of B {:.4f}".format(
CRIT, Q))
print("logged visitors false alarm (no fault) fault detected")
for n in [2000, 10000, 50000, 200000]:
print("{:>15,} {:>24.4f} {:>16.3f}".format(n, rate(n, 0.5), rate(n, Q)))
p_true = 0.050
print("null variant: true pay rate {:.3f} on both arms; B measured "
"{:.4f}".format(p_true, p_true / (1 - LOSS)))
print(" phantom lift {:+.1%} from the fault alone".format(
1 / (1 - LOSS) - 1))Single source and citation
Fan, Y. (2026). Autonomous Nightly Funnel Micro-Surgery, Bandit Traffic Allocation and Zero-Touch Conversion Self-Improvement (Version 1.0.0), Chapter 6: Trustworthy Data: Negative Bias in Adaptive Data, Sample Ratio Mismatch and Guardrail Rollback. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-405 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026405,
author = {Fan, Yafu},
title = {Autonomous Nightly Funnel Micro-Surgery, Bandit Traffic Allocation and Zero-Touch Conversion Self-Improvement},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · The Funnel Check-up: Six-Step Beta Posteriors, Nightly Change Priorities and a Compliant Candidate Pool
- Chapter 2 · Thompson Sampling and Regret: from Two Small Samples in 1933 to the Even Split
- Chapter 3 · The Reward Function: Net Contribution per Visitor, Not the Conversion Rate
- Chapter 4 · Non-stationary Traffic: Weekday Effects, Discounted Memory and Sliding Windows
- Chapter 5 · Elimination Rules: Minimum Exposure, Posterior Win Probability and Time-uniform Confidence Sequences
- Chapter 7 · Classical Sources East and West: "At the Limit, Change" in the Great Commentary, the Sunzi on Weak and Strong Points, and Random Allocation in Peirce and Jastrow
- Chapter 8 · The Nightly Funnel Ledger: Field Tools and Checker Specifications, a Pre-registered Test and the Conditions of Refutation