What does Four-Player Imperfect Information conclude, in numbers, about “Practice and a Pre-Registered Test”?

Answer: Chapter 8 gives a four-step checklist and a pre-registered duplicate-design policy comparison scored at the first differing decision: smallest effect of interest 0.010 buy-ins per scored decision, 12,000 scored decisions, the decision rule written as code; the conclusion holds only against the fixed opponent pool used. Marked boxes separate published evidence, runnable Python calculations (16 programs), classical sources and the author's models and hypotheses; apart from cited literature data, all data in the book are synthetic.

Scope of the numbers: Marked boxes separate published evidence, runnable Python calculations (16 programs), classical sources and the author's models and hypotheses; apart from cited literature data, all data in the book are synthetic.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.010, 12000

Duplicate design: the same deals replayed under two policies, scored at
the first differing decision, against a fixed opponent pool.
SESOI 0.010 buy-ins per scored decision; paired s.d. 0.120.
normal theory: n = 1,513.1 -> 1,514 scored decisions
generator's actual s.d.: 0.1256 (the planning value was 0.120)
normal theory at the generator's s.d.: 1,657 scored decisions
simulation with the heavy-tailed paired difference:
   n =  1,657: power 0.880 (s.e. 0.013)
   n = 12,000: power 1.000 (s.e. 0.000)
   n = 15,000: power 1.000 (s.e. 0.000)
   n = 18,000: power 1.000 (s.e. 0.000)
   smallest n with power - 2 s.e. >= 0.90: 12,000
   one-sided error rate with no effect: 0.033 (nominal 0.025)
planned: 12,000 scored decisions. At roughly one scored decision per hand
and 60 hands per session, that is about 200 sessions of play, or
a few weeks of continuous self-play for a software agent.
sensitivity (normal theory):
   SESOI \ paired s.d.      0.08      0.12      0.20
   0.005 buy-ins            2,690     6,053    16,812
   0.010 buy-ins              673     1,514     4,203
   0.020 buy-ins              169       379     1,051
Two things this sizing makes explicit. First, a human cannot run this
trial: 12,000 scored decisions is beyond what anyone plays deliberately, so
the comparison belongs to software agents replaying stored deals, and the
conclusions then apply to the policies rather than to people. Second, the
SESOI of 0.010 buy-ins per decision is small in absolute terms and still
needs this many observations, which is the quantitative reason the strategy
debates in this game remain unsettled: the sample sizes required have
Show the full program
# Chapter 8. Sample size for the pre-registered policy comparison. Design: a
# duplicate design in which the SAME deals are replayed with two decision
# policies, scored at the first decision at which the policies differ, against
# a fixed opponent pool. Primary
# outcome: the mean points difference per scored decision. The pairing is what
# makes the trial feasible (p11); the program sizes it on the paired standard
# deviation and then checks the normal-theory answer against a simulation with
# a realistic, heavy-tailed outcome distribution, because mahjong scores are
# not normal and the t test's behaviour in the tail matters.
import math
import random
from statistics import NormalDist

Z = NormalDist().inv_cdf
SESOI = 0.010           # buy-ins per scored decision
SD_DIFF = 0.120         # paired s.d. from the duplicate design (p11)

def n_paired(d=SESOI, sd=SD_DIFF, power=0.90, alpha=0.05):
    return (Z(1 - alpha / 2) + Z(power)) ** 2 * sd ** 2 / d ** 2

# a heavy-tailed paired difference: mostly zero, occasionally a large swing
def draw_diff(rng, delta):
    u = rng.random()
    if u < 0.70:
        return delta + rng.gauss(0.0, 0.02)          # small or no consequence
    if u < 0.94:
        return delta + rng.gauss(0.0, 0.12)          # a normal-sized swing
    # a big hand changes hands
    return delta + rng.gauss(0.0, 0.45)

def one_trial(rng, n, delta):
    xs = [draw_diff(rng, delta) for _ in range(n)]
    m = math.fsum(xs) / n
    v = math.fsum((x - m) ** 2 for x in xs) / (n - 1)
    se = math.sqrt(v / n)
    t = m / se if se > 0 else 0.0
    return t > Z(0.975), m

if __name__ == "__main__":
    print(
        "Duplicate design: the same deals replayed under two policies,"
        " scored at"
    )
    print("the first differing decision, against a fixed opponent pool.")
    print("SESOI {:.3f} buy-ins per scored decision; paired s.d. {:.3f}.\n"
          .format(SESOI, SD_DIFF))

    n0 = n_paired()
    print("normal theory: n = {:,.1f} -> {:,} scored decisions"
          .format(n0, math.ceil(n0)))

    rng = random.Random(5514)
    # empirical s.d. of the heavy-tailed generator, for comparison
    samp = [draw_diff(rng, 0.0) for _ in range(200000)]
    m = math.fsum(samp) / len(samp)
    sd_emp = math.sqrt(
        math.fsum((x - m) ** 2 for x in samp) / (len(samp) - 1)
    )
    print("generator's actual s.d.: {:.4f} (the planning value was {:.3f})"
          .format(sd_emp, SD_DIFF))
    n_adj = n_paired(sd=sd_emp)
    print("normal theory at the generator's s.d.: {:,} scored decisions"
          .format(math.ceil(n_adj)))

    print("\nsimulation with the heavy-tailed paired difference:")
    chosen = None
    for n in (math.ceil(n_adj), 12000, 15000, 18000):
        res = [one_trial(rng, n, SESOI) for _ in range(600)]
        pw = sum(r[0] for r in res) / len(res)
        se = math.sqrt(pw * (1 - pw) / len(res))
        print("   n = {:6,}: power {:.3f} (s.e. {:.3f})".format(n, pw, se))
        if chosen is None and pw - 2 * se >= 0.90:
            chosen = n
    if chosen is None:
        chosen = 18000
    t1 = sum(one_trial(rng, chosen, 0.0)[0] for _ in range(600)) / 600
    print("   smallest n with power - 2 s.e. >= 0.90: {:,}".format(chosen))
    print(
        "   one-sided error rate with no effect: {:.3f} (nominal 0.025)"
        .format(t1)
    )

    print(
        "\nplanned: {:,} scored decisions. At roughly one scored decision per"
        " hand".format(chosen)
    )
    print(
        "and 60 hands per session, that is about {:,.0f} sessions of play, or"
        .format(chosen / 60)
    )
    print("a few weeks of continuous self-play for a software agent.")

    print("\nsensitivity (normal theory):")
    print("   SESOI \\ paired s.d.      0.08      0.12      0.20")
    for d in (0.005, 0.010, 0.020):
        print("   {:.3f} buy-ins       ".format(d) + "".join(
            "{:10,.0f}".format(math.ceil(n_paired(d=d, sd=s)))
            for s in (0.08, 0.12, 0.20)))

    print(
        "\nTwo things this sizing makes explicit. First, a human cannot run"
        " this"
    )
    print(
        "trial: {:,} scored decisions is beyond what anyone plays"
        " deliberately, so".format(chosen)
    )
    print(
        "the comparison belongs to software agents replaying stored deals,"
        " and the"
    )
    print(
        "conclusions then apply to the policies rather than to people."
        " Second, the"
    )
    print(
        "SESOI of {:.3f} buy-ins per decision is small in absolute terms and"
        " still".format(SESOI)
    )
    print(
        "needs this many observations, which is the quantitative reason the"
        " strategy"
    )
    print(
        "debates in this game remain unsettled: the sample sizes required"
        " have"
    )
    print("rarely been assembled.")

Single source and citation

Fan, Y. (2026). Imperfect-Information Stochastic Games, Bayesian Hand-Reading and Counterfactual Regret Minimisation in Four-Player Tile Dynamics (Version 1.0.0), Chapter 8: Practice and a Pre-Registered Test. 5D Library. https://doi.org/10.5281/zenodo.24020464

DOI: 10.5281/zenodo.24020464 · Read the monograph · 5DL-B-055 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026055,
  author    = {Fan, Yafu},
  title     = {Imperfect-Information Stochastic Games, Bayesian Hand-Reading and Counterfactual Regret Minimisation in Four-Player Tile Dynamics},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.24020464},
  url       = {https://doi.org/10.5281/zenodo.24020464}
}

Other chapter answers in this book