What does Zero-Touch Sites, Payment Anti-Fraud and Cashflow Queues conclude, in numbers, about “The Order State Machine: Exhaustive Checking and the Idempotency Window”?
Answer: Chapter 3's exhaustive model check shows that handlers blind to state double-release with two events, and that idempotency keys must outlive the 511-minute retry schedule and be written atomically.
Scope of the numbers: Marked boxes separate evidence, 16 runnable Python programs, historical sources and hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 511
retry schedule spans 511 minutes
copies delivered beyond the first: 26,883
race copies within 0.2 s: 486
TTL (min) processed twice, plain with atomic check
0 26,883 26,883
10 9,912 9,426
60 3,015 2,529
600 486 0
1440 486 0
Reading: retries are the normal case, not the exception:
tens of thousands of copies per 100,000 orders, spread by a
slow-down over hours of back-off. A key kept for ten minutes
or an hour still lets thousands through; only a TTL longer
than the whole schedule together with an atomic check-and-
insert stops them all.Show the full program
# Chapter 3. [p06] At-least-once notices and the idempotency window.
# A payment notice (webhook) is retried until acknowledged: the sender
# waits 1, 2, 4, ..., 256 minutes between tries. A delivery that was
# processed is retried anyway when its acknowledgement is lost: with
# chance 0.03 at random, and throughout a receiver slow-down (chance
# 0.05 per order, lasting an exponential time of mean 30 minutes)
# during which work is done but every acknowledgement times out. With
# chance 0.005 a proxy also resends the first copy after about 0.05 s.
# The receiver keeps each idempotency key for a time-to-live (TTL); a
# copy arriving after the key has expired is processed again. Two
# copies within the 0.2 s it takes to process one notice both pass the
# key check unless the check-and-insert is atomic.
# 100,000 orders are simulated; duplicates processed per 100,000 are
# reported for each TTL, without and with an atomic check.
import random
GAPS = [2 ** i for i in range(9)]
def deliveries(rng):
slow = rng.expovariate(1 / 30) if rng.random() < 0.05 else -1.0
t, out = 0.0, []
for g in [0] + GAPS:
t += g
out.append(t)
if t >= slow and rng.random() >= 0.03:
break
if rng.random() < 0.005:
out.insert(1, rng.expovariate(1 / 0.05) / 60)
return out
if __name__ == "__main__":
rng = random.Random(75006)
orders = [deliveries(rng) for _ in range(100000)]
dup = sum(len(o) - 1 for o in orders if o)
print(" retry schedule spans {} minutes".format(sum(GAPS)))
print(" copies delivered beyond the first: {:,d}".format(dup))
print(" race copies within 0.2 s: {:,d}".format(sum(
1 for o in orders for a, b in zip(o, o[1:]) if b - a < 0.2 / 60)))
print(" TTL (min) processed twice, plain with atomic check")
for ttl in (0, 10, 60, 600, 1440):
plain = atomic = 0
for o in orders:
if not o:
continue
first = o[0]
for t in o[1:]:
late = t - first > ttl
race = t - first < 0.2 / 60 and not late
plain += late or race
atomic += late
print(" {:9d} {:24,d} {:19,d}".format(ttl, plain, atomic))
print("\nReading: retries are the normal case, not the exception:")
print("tens of thousands of copies per 100,000 orders, spread by a")
print("slow-down over hours of back-off. A key kept for ten minutes")
print("or an hour still lets thousands through; only a TTL longer")
print("than the whole schedule together with an atomic check-and-")
print("insert stops them all.")Single source and citation
Fan, Y. (2026). Zero-Touch Multi-Site Fault Tolerance, Payment Anti-Fraud State Machines and Passive Cashflow Queuing (Version 1.0.0), Chapter 3: The Order State Machine: Exhaustive Checking and the Idempotency Window. 5D Library. https://doi.org/10.5281/zenodo.24090201
DOI: 10.5281/zenodo.24090201 · Read the monograph · 5DL-B-075 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026075,
author = {Fan, Yafu},
title = {Zero-Touch Multi-Site Fault Tolerance, Payment Anti-Fraud State Machines and Passive Cashflow Queuing},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.24090201},
url = {https://doi.org/10.5281/zenodo.24090201}
}Other chapter answers in this book
- Chapter 1 · Transfer Receipts: Deduplication, Image Hashes and Release-Then-Audit
- Chapter 2 · Self-Referral and "Invite Friends to Unlock"
- Chapter 4 · Thirty Sites on One Kernel: Queues and Tail Latency
- Chapter 5 · Availability: Shared Kernels, Standby Nodes and Blue-Green Draining
- Chapter 6 · Allocating Traffic, Promoting Domains and Holding Cash
- Chapter 7 · Matched Tallies, Banknote Seals and Liu Yan's Canal
- Chapter 8 · A Nine-Flag Audit and a Pre-Registered Experiment