What does Conformal Intervals for Orders conclude, in numbers, about “How Large a Calibration Set: Coverage Given the Calibration Set Follows a Beta Law”?
Answer: Chapter 4 shows that coverage given a calibration set follows a Beta law; with twenty calibration values the chance that true coverage is below 0.90 is 0.1216.
Scope of the numbers: Parameters are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (23 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.1216, 0.90
| alpha = 0.05; coverage | calibration ~ Beta(k, m + 1 - k) | ||||||
| m | k | mean | 5% pt | 95% pt | P(<0.90) | P(<0.93) |
| 19 | 19 | 0.9500 | 0.8541 | 0.9973 | 0.1351 | 0.2519 |
| 20 | 20 | 0.9524 | 0.8609 | 0.9974 | 0.1216 | 0.2342 |
| 50 | 49 | 0.9608 | 0.9086 | 0.9928 | 0.0338 | 0.1265 |
| 100 | 96 | 0.9505 | 0.9108 | 0.9801 | 0.0237 | 0.1632 |
| 200 | 191 | 0.9502 | 0.9228 | 0.9726 | 0.0035 | 0.1010 |
| 500 | 476 | 0.9501 | 0.9332 | 0.9650 | 0.0000 | 0.0280 |
| 1000 | 951 | 0.9500 | 0.9382 | 0.9608 | 0.0000 | 0.0040 |
| 2000 | 1901 | 0.9500 | 0.9418 | 0.9578 | 0.0000 | 0.0001 |
Show the full program
# Chapter 4. p41: coverage given one calibration set is a Beta draw.
# Model: with m continuous exchangeable scores and rank
# k = ceil((m + 1)(1 - alpha)), the coverage conditional on the
# calibration set follows Beta(k, m + 1 - k). Its CDF at x equals
# P(Binomial(m, x) >= k), summed exactly with math.fsum; quantiles
# come from bisection. Nothing here is simulated.
import math
def beta_cdf(x, k, m):
if x <= 0.0:
return 0.0
if x >= 1.0:
return 1.0
lx, l1 = math.log(x), math.log1p(-x)
terms = [math.exp(math.lgamma(m + 1) - math.lgamma(j + 1)
- math.lgamma(m - j + 1) + j * lx + (m - j) * l1)
for j in range(k, m + 1)]
return math.fsum(terms)
def beta_q(p, k, m):
lo, hi = 0.0, 1.0
for _ in range(60):
mid = (lo + hi) / 2
if beta_cdf(mid, k, m) < p:
lo = mid
else:
hi = mid
return (lo + hi) / 2
if __name__ == "__main__":
a = 0.05
print("alpha = 0.05; coverage | calibration ~ Beta(k, m + 1 - k)")
print(" m k mean 5% pt 95% pt P(<0.90) P(<0.93)")
for m in (19, 20, 50, 100, 200, 500, 1000, 2000):
k = math.ceil((m + 1) * (1 - a))
print("{:5d} {:5d} {:.4f} {:.4f} {:.4f} {:.4f} {:.4f}"
.format(m, k, k / (m + 1), beta_q(0.05, k, m),
beta_q(0.95, k, m), beta_cdf(0.90, k, m),
beta_cdf(0.93, k, m)))Single source and citation
Fan, Y. (2026). Conformal Prediction and Distribution-Free Intervals: Finite-Sample Coverage Guarantees for Order Counts and Order Values (Version 1.0.0), Chapter 4: How Large a Calibration Set: Coverage Given the Calibration Set Follows a Beta Law. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-306 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026306,
author = {Fan, Yafu},
title = {Conformal Prediction and Distribution-Free Intervals: Finite-Sample Coverage Guarantees for Order Counts and Order Values},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · The Coverage Gap of Normal Intervals: Heavy-Tailed Orders Push Every Miss to the Right
- Chapter 2 · The Correct Rank: Why "Plus One" Matters and Where the Finite-Sample Guarantee Comes From
- Chapter 3 · The Price of the Guarantee: Interval Width, the Shape of the Score and the Extra Paid When Normality Holds
- Chapter 5 · Marginal Is Not Conditional: Traffic Bands, Customer Segments and Intervals Too Wide to Use
- Chapter 6 · When Exchangeability Breaks: Weekend Effects, Growth Drift, Promotion Shocks and Three Repairs
- Chapter 7 · Classical Sources East and West: Guo Shoujing's "The Calendar Rests on Measurement" and Neyman's Confidence Intervals
- Chapter 8 · Closing the Loop: the Weekly Order-Count Interval Sheet, the Conformal Interval Calculator and a Pre-Registered Test