What does Conformal Intervals for Orders conclude, in numbers, about “How Large a Calibration Set: Coverage Given the Calibration Set Follows a Beta Law”?

Answer: Chapter 4 shows that coverage given a calibration set follows a Beta law; with twenty calibration values the chance that true coverage is below 0.90 is 0.1216.

Scope of the numbers: Parameters are illustrative; all calculations are generated by the book's programs; the book is general business analysis, not individualised business advice. Marked boxes separate published evidence, runnable Python calculations (23 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.1216, 0.90

alpha = 0.05; coverage | calibration ~ Beta(k, m + 1 - k)
mkmean5% pt95% ptP(<0.90)P(<0.93)
19190.95000.85410.99730.13510.2519
20200.95240.86090.99740.12160.2342
50490.96080.90860.99280.03380.1265
100960.95050.91080.98010.02370.1632
2001910.95020.92280.97260.00350.1010
5004760.95010.93320.96500.00000.0280
10009510.95000.93820.96080.00000.0040
200019010.95000.94180.95780.00000.0001
Show the full program
# Chapter 4. p41: coverage given one calibration set is a Beta draw.
# Model: with m continuous exchangeable scores and rank
# k = ceil((m + 1)(1 - alpha)), the coverage conditional on the
# calibration set follows Beta(k, m + 1 - k). Its CDF at x equals
# P(Binomial(m, x) >= k), summed exactly with math.fsum; quantiles
# come from bisection. Nothing here is simulated.
import math


def beta_cdf(x, k, m):
    if x <= 0.0:
        return 0.0
    if x >= 1.0:
        return 1.0
    lx, l1 = math.log(x), math.log1p(-x)
    terms = [math.exp(math.lgamma(m + 1) - math.lgamma(j + 1)
                      - math.lgamma(m - j + 1) + j * lx + (m - j) * l1)
             for j in range(k, m + 1)]
    return math.fsum(terms)


def beta_q(p, k, m):
    lo, hi = 0.0, 1.0
    for _ in range(60):
        mid = (lo + hi) / 2
        if beta_cdf(mid, k, m) < p:
            lo = mid
        else:
            hi = mid
    return (lo + hi) / 2


if __name__ == "__main__":
    a = 0.05
    print("alpha = 0.05; coverage | calibration ~ Beta(k, m + 1 - k)")
    print("    m     k   mean    5% pt   95% pt  P(<0.90)  P(<0.93)")
    for m in (19, 20, 50, 100, 200, 500, 1000, 2000):
        k = math.ceil((m + 1) * (1 - a))
        print("{:5d} {:5d}   {:.4f}  {:.4f}  {:.4f}  {:.4f}    {:.4f}"
              .format(m, k, k / (m + 1), beta_q(0.05, k, m),
                      beta_q(0.95, k, m), beta_cdf(0.90, k, m),
                      beta_cdf(0.93, k, m)))

Single source and citation

Fan, Y. (2026). Conformal Prediction and Distribution-Free Intervals: Finite-Sample Coverage Guarantees for Order Counts and Order Values (Version 1.0.0), Chapter 4: How Large a Calibration Set: Coverage Given the Calibration Set Follows a Beta Law. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-306 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026306,
  author    = {Fan, Yafu},
  title     = {Conformal Prediction and Distribution-Free Intervals: Finite-Sample Coverage Guarantees for Order Counts and Order Values},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book