5D Library › Prediction Market Microstructure, Bregman Information Projection and Cross-Book Arbitrage Bounds · 中文

Chapter 6. Reading Probabilities from Prices: Calibration Tests, Correlated Resolutions and the Cost of Correction

The bias exists — so what?

The first five chapters showed why prices depart from probabilities: dispersed beliefs, risk attitudes, capital lock-up, arbitrage that cannot be executed. A natural next step is to correct a systematic bias — to read a 0.05 longshot as 0.033, say. This chapter asks three questions. How many resolved contracts does it take to confirm the bias? Are those contracts really independent observations? And does correcting the prices risk making the forecasts worse?

How much data it takes to see the bias

Calculation

A one-parameter calibration model. Let the true log-odds be β\beta times the log-odds of the price: logit⁡p=βlogit⁡π\operatorname{logit} p = \beta \operatorname{logit} \pi. β=1\beta = 1 is perfect calibration; β>1\beta > 1 means longshots overpriced and favourites underpriced. Estimate β\beta by maximum likelihood from the outcomes (0 or 1) of resolved contracts and test β=1\beta = 1.

Calculation

# Chapter 6. p12: How many resolved contracts does it take to see the bias?
# Prices are uniform on the log-odds scale in [-3, 3]. The truth is
#     logit(p) = beta * logit(price),   beta = 1.15
# (beta > 1: longshots overpriced, favourites underpriced; at a price of
# 0.05 the true probability is 0.033). For each sample size, 200 samples
# are drawn; beta is fitted by maximum likelihood (Newton) and the test
# of beta = 1 at the 5 % level is counted. A calibrated market (beta = 1)
# is also run to check the false-positive rate.
import math
import random

def sig(x):
    return 1 / (1 + math.exp(-x))

def fit(xs, ys):
    b = 1.0
    for _ in range(25):
        g = h = 0.0
        for x, y in zip(xs, ys):
            p = sig(b * x)
            g += (y - p) * x
            h += p * (1 - p) * x * x
        b += g / h
        if abs(g / h) < 1e-10:
            break
    return b, 1 / math.sqrt(h)

def run(rng, n, beta, reps=200):
    hits, ests = 0, []
    for _ in range(reps):
        xs = [rng.uniform(-3, 3) for _ in range(n)]
        ys = [1 if rng.random() < sig(beta * x) else 0 for x in xs]
        b, se = fit(xs, ys)
        ests.append(b)
        hits += abs(b - 1) / se > 1.96
    m = sum(ests) / reps
    sd = math.sqrt(sum((e - m) ** 2 for e in ests) / (reps - 1))
    return hits / reps, m, sd

if __name__ == "__main__":
    rng = random.Random(59012)
    print("price 0.05 -> true p {:.3f} when beta = 1.15".format(
        sig(1.15 * math.log(0.05 / 0.95))))
    print("\n     n   power (beta=1.15)  mean est   sd    "
          "false pos (beta=1)")
    for n in (100, 300, 1000, 3000):
        pw, m, sd = run(rng, n, 1.15)
        fp, _, _ = run(rng, n, 1.0)
        print("  {:5d}        {:5.3f}          {:5.3f}   {:5.3f}"
              "       {:5.3f}".format(n, pw, m, sd, fp))
    print("\nReading: a bias that turns a 5 % price into a 3.3 % chance is")
    print("invisible in 100 resolved contracts, a coin flip at 1,000 and")
    print("reliably seen only at 3,000 - and that assumes the contracts")
    print("resolve independently, which the next program questions.")
# price 0.05 -> true p 0.033 when beta = 1.15
#
#      n   power (beta=1.15)  mean est   sd    false pos (beta=1)
#     100        0.065          1.196   0.253       0.055
#     300        0.255          1.176   0.129       0.055
#    1000        0.575          1.151   0.074       0.035
#    3000        0.985          1.153   0.036       0.045
#
# Reading: a bias that turns a 5 % price into a 3.3 % chance is
# invisible in 100 resolved contracts, a coin flip at 1,000 and
# reliably seen only at 3,000 - and that assumes the contracts
# resolve independently, which the next program questions.

At β=1.15\beta = 1.15 a contract priced at 0.05 has a true probability of 0.033. That is a material bias: a unit bet on longshots returns only two thirds on average. Yet with 100 resolved contracts the test has a power of only 0.065, barely above its false-positive rate; with 300, 0.255; with 1,000, 0.575, a coin flip; only at 3,000 does it reach 0.985. The false-positive rate stays near 5 % at every sample size, so the test is sound — the data are too few.

Contracts are not independent

That calculation assumes each contract resolves independently. Real prediction markets do not work that way: one election lists dozens of contracts at once — states, vote-share brackets, turnout — all decided by the same ballots. The thirteenth program builds a market whose prices are exactly right: each contract’s price is its true probability. The contracts come from 20 events of 50 each, and outcomes within an event share a shock (a Gaussian copula with correlation ρ\rho) that leaves every contract’s marginal probability unchanged.

Calculation

# Chapter 6. p13: Correlated resolutions shrink the effective sample.
# 1,000 contracts come from 20 events (50 per event: states, margins,
# turnout lines of one election, say). Prices are perfectly calibrated:
# each contract's price equals its true probability, drawn uniform in
# [0.80, 0.97]. Outcomes share an event-level shock through a Gaussian
# copula with correlation rho, which leaves every marginal probability
# unchanged. The calibration estimate is mean(outcome - price); the
# naive standard error treats contracts as independent. 2,000 runs.
import math
import random
from statistics import NormalDist

E, M, RUNS = 20, 50, 2000
PHI = NormalDist()

def one(rng, rho):
    total = var = 0.0
    for _ in range(E):
        z = rng.gauss(0, 1)
        for _ in range(M):
            p = rng.uniform(0.80, 0.97)
            u = PHI.cdf(math.sqrt(rho) * z + math.sqrt(1 - rho)
                        * rng.gauss(0, 1))
            total += (1 if u < p else 0) - p
            var += p * (1 - p)
    n = E * M
    return total / n, math.sqrt(var) / n

if __name__ == "__main__":
    rng = random.Random(59013)
    print("   rho   true sd   naive se   ratio   effective n"
          "   naive false positives")
    for rho in (0.0, 0.1, 0.3, 0.6):
        est, ses = [], []
        for _ in range(RUNS):
            e, s = one(rng, rho)
            est.append(e)
            ses.append(s)
        sd = math.sqrt(sum(x * x for x in est) / RUNS)
        se = sum(ses) / RUNS
        fp = sum(abs(e) / s > 1.96 for e, s in zip(est, ses)) / RUNS
        print("   {:3.1f}   {:7.4f}    {:7.4f}   {:5.2f}      {:6.0f}"
              "          {:6.3f}".format(rho, sd, se, sd / se,
                                     E * M * (se / sd) ** 2, fp))
    print("\nReading: the prices are right by construction, yet with an")
    print("event-level correlation of 0.3 a naive test calls them biased")
    print("in about half of all samples. 1,000 contracts from 20 events")
    print("carry the information of far fewer; inference must cluster by")
    print("event.")
#    rho   true sd   naive se   ratio   effective n   naive false positives
#    0.0    0.0098     0.0100    0.98        1036           0.044
#    0.1    0.0170     0.0100    1.70         344           0.243
#    0.3    0.0267     0.0100    2.68         139           0.470
#    0.6    0.0401     0.0100    4.02          62           0.628
#
# Reading: the prices are right by construction, yet with an
# event-level correlation of 0.3 a naive test calls them biased
# in about half of all samples. 1,000 contracts from 20 events
# carry the information of far fewer; inference must cluster by
# event.

Without correlation the naive standard error is right and the false-positive rate is 0.044. With a within-event latent (copula) correlation ρ\rho of 0.3, the true standard deviation of the estimate is 2.68 times the naive standard error, 1,000 contracts are worth only 139 independent ones, and a naive test declares the prices biased in 0.470 of samples — although they are exactly right. At a correlation of 0.6 the effective sample is down to 62 contracts and the false-positive rate is 0.628.

Effective sample size of 1,000 correctly priced contracts against the within-event correlation (simulated as in p13); the more contracts per event, the faster it shrinks.

Testing the calibration of a prediction market therefore requires standard errors clustered by event (Cameron and Miller 2015), and sample sizes planned in events, not contracts. A study that uses “thousands of contracts” to show that a platform is, or is not, systematically biased may, without clustering, rest on a few dozen events.

The cost of correcting the bias

Suppose the bias is real. Is correcting it worth it? The fourteenth program first uses the Murphy decomposition (Murphy 1973) of the Brier score (Brier 1950) to see how large the bias is, then compares two corrections out of sample.

Calculation

# Chapter 6. p14: Correcting a biased price: when it helps and when it hurts.
# Prices are uniform on log-odds in [-3, 3]; truth logit(p) = 1.15 x
# logit(price), as in p12. Part A splits the market's Brier score into
# reliability (miscalibration, smaller is better), resolution (larger is
# better) and uncertainty (Murphy 1973), 10 price bins, 200,000 contracts.
# Part B learns a correction from n_train resolved contracts and scores
# it on 50,000 fresh ones, 100 repetitions:
#   slope   refit beta in logit(p) = beta logit(price)   (1 parameter)
#   bins    replace each price by its bin's training frequency (10 bins)
# Reported: mean change in out-of-sample Brier score (negative = better).
import math
import random

def sig(x):
    return 1 / (1 + math.exp(-x))

def draw(rng, n):
    xs = [rng.uniform(-3, 3) for _ in range(n)]
    return [(sig(x), 1 if rng.random() < sig(1.15 * x) else 0) for x in xs]

def bin_of(p):
    return min(int(p * 10), 9)

def murphy(rows):
    n = len(rows)
    base = sum(y for _, y in rows) / n
    rel = res = 0.0
    for k in range(10):
        g = [(p, y) for p, y in rows if bin_of(p) == k]
        if not g:
            continue
        fp = sum(p for p, _ in g) / len(g)
        fy = sum(y for _, y in g) / len(g)
        rel += len(g) * (fp - fy) ** 2 / n
        res += len(g) * (fy - base) ** 2 / n
    brier = sum((p - y) ** 2 for p, y in rows) / n
    return brier, rel, res, base * (1 - base)

def fit_slope(rows):
    b = 1.0
    for _ in range(25):
        g = h = 0.0
        for p, y in rows:
            x = math.log(p / (1 - p))
            q = sig(b * x)
            g += (y - q) * x
            h += q * (1 - q) * x * x
        b += g / h
    return b

def brier(rows, f):
    return sum((f(p) - y) ** 2 for p, y in rows) / len(rows)

if __name__ == "__main__":
    rng = random.Random(59014)
    b, rel, res, unc = murphy(draw(rng, 200000))
    print("A. Brier {:.5f}; reliability {:.5f}; resolution {:.5f}"
          .format(b, rel, res))
    print("   uncertainty {:.5f}; rel - res + unc = {:.5f}"
          " (rest: within-bin spread)".format(unc, rel - res + unc))
    test = draw(rng, 50000)
    raw = brier(test, lambda p: p)
    print("\nB. raw out-of-sample Brier {:.5f}".format(raw))
    print("   n_train   slope fix   bins fix    (change in Brier)")
    for n in (30, 100, 300, 1000, 10000):
        ds = db = 0.0
        for _ in range(100):
            tr = draw(rng, n)
            beta = fit_slope(tr)
            freq = {}
            for k in range(10):
                g = [y for p, y in tr if bin_of(p) == k]
                freq[k] = sum(g) / len(g) if g else (k + 0.5) / 10
            ds += brier(test, lambda p: sig(beta * math.log(p / (1 - p))))
            db += brier(test, lambda p: freq[bin_of(p)])
        print("   {:7d}   {:+.5f}   {:+.5f}".format(
            n, ds / 100 - raw, db / 100 - raw))
    print("\nReading: the bias is real but small - reliability is a tiny")
    print("part of the Brier score. A one-parameter correction pays off")
    print("once a few hundred contracts are resolved and hurts below that.")
    print("A ten-bin correction is worse at every size tried: it throws")
    print("away the resolution inside each bin to fix a small bias.")
# A. Brier 0.13714; reliability 0.00053; resolution 0.11262
#    uncertainty 0.25000; rel - res + unc = 0.13791 (rest: within-bin spread)
#
# B. raw out-of-sample Brier 0.13675
#    n_train   slope fix   bins fix    (change in Brier)
#         30   +0.00289   +0.05731
#        100   +0.00038   +0.01960
#        300   -0.00021   +0.00555
#       1000   -0.00052   +0.00177
#      10000   -0.00057   +0.00035
#
# Reading: the bias is real but small - reliability is a tiny
# part of the Brier score. A one-parameter correction pays off
# once a few hundred contracts are resolved and hurts below that.
# A ten-bin correction is worse at every size tried: it throws
# away the resolution inside each bin to fix a small bias.

Part A splits the Brier score of 200,000 contracts, 0.13714, into reliability 0.00053 (the calibration error), resolution 0.11262 and uncertainty 0.25000. The β=1.15\beta = 1.15 bias is real, but it is a small part of the total: most of a price’s value lies in separating likely events from unlikely ones, and the bias does not touch that.

Part B learns a correction from nn training contracts and scores it on 50,000 fresh ones. The one-parameter slope correction makes the Brier score worse at 30 and 100 contracts (+0.00289 and +0.00038), starts to help from 300, and at 10,000 improves it by −0.00057 — close to the most the bias allows. The ten-bin correction is worse at every sample size: by +0.05731 at 30 and still by +0.00035 at 10,000, because it replaces each price with its bin’s average frequency, throwing away the resolution inside each bin to fix a small bias.

Note

The scope of the result. These numbers come from a simulated market known to have only a slope bias. Real biases may vary with the horizon (Chapter 3), the platform and the type of event, which needs more parameters and so more data. The conclusion here is directional: the more flexible the correction, the more resolved contracts it needs, and the more likely it is to do harm when data are short.

Hypothesis · not established

The author’s hypothesis (not yet tested): among published studies claiming that a prediction market is “well calibrated” or “systematically biased”, a substantial share lose significance once standard errors are recomputed with clustering by event. What would show it wrong: after reanalysis, almost all conclusions keep their significance.

Summary

The bias in prices is real, but small enough to need a great deal of data to confirm: a bias that turns a 5 % price into a 3.3 % chance is detected with only about even odds in 1,000 independent contracts. And contracts are not independent: with a within-event correlation of 0.3, 1,000 contracts are worth 139, and a naive test falsely reports a bias in nearly half of all samples. The bias is a small part of the Brier score; a one-parameter correction helps only after a few hundred contracts, and a flexible binned correction hurts at every sample size in the example.

References

Brier, Glenn W. 1950. “Verification of Forecasts Expressed in Terms of Probability.” Monthly Weather Review 78 (1): 1–3.
Cameron, A. Colin, and Douglas L. Miller. 2015. “A Practitioner’s Guide to Cluster-Robust Inference.” Journal of Human Resources 50 (2): 317–72.
Murphy, Allan H. 1973. “A New Vector Partition of the Probability Score.” Journal of Applied Meteorology 12 (4): 595–600.

Fan, Y. (2026). Prediction Market Microstructure, Bregman Information Projection and Cross-Book Arbitrage Bounds: Logarithmic Market Scoring Rules, Favorite-Longshot Calibration Bias, Lock-Up Opportunity Cost and Frank-Wolfe Combinatorial Pricing [预测市场微观结构、布雷格曼信息投影与跨盘口无套利边界] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090166 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).