5D Library › Imperfect-Information Stochastic Games, Bayesian Hand-Reading and Counterfactual Regret Minimisation in Four-Player Tile Dynamics · 中文

Chapter 5. Folding: Optimal Stopping, and a Conclusion Stated Backwards

Written as optimal stopping

One opponent is visibly ready. From then on, each turn the player either pushes (discards a tile that may deal in, keeping their own chance alive) or folds (discards only safe tiles thereafter, abandoning their hand but no longer able to deal in).

This is a finite-horizon Markov decision process (Puterman 1994), solvable exactly by backward induction (Bellman 1957); the general theory of optimal stopping is in Chow, Robbins and Siegmund (Chow, Robbins, and Siegmund 1971), and Volume 42 covered this class of problem.

The key modelling choice: folding is absorbing. Once a player folds they stay folded for the rest of the hand, so the value of folding is just the opponent’s self-draw risk over the remaining turns.

Calculation

# Chapter 5. Folding as optimal stopping, and the limit of that framing. With
# one opponent visibly ready, each remaining turn a player either PUSHES
# (discards a tile that may deal in, keeping their own chance alive) or FOLDS
# (discards only safe tiles for the rest of the hand, abandoning their own
# hand but no longer able to deal in). Folding is absorbing: once a player
# folds they stay folded, so the value of folding at turn t is just the
# opponent's self-draw risk over the remaining turns. Pushing is solved by
# backward induction against that. Illustrative points; standard library only.
import math

def fold_value(t, turns, p_tsumo, v_tsumo):
    """Fold now and stay folded: lose v_tsumo if the opponent self-draws at
    any
    remaining turn."""
    remaining = turns - t
    return -(1.0 - (1.0 - p_tsumo) ** remaining) * v_tsumo

def solve(turns, p_win, v_win, p_deal, v_deal, p_tsumo, v_tsumo):
    V = [0.0] * (turns + 1)
    policy = [None] * turns
    for t in range(turns - 1, -1, -1):
        # push this turn: own win, deal-in, opponent self-draw, or continue
        p_cont = 1.0 - p_win - p_deal - p_tsumo
        push = (p_win * v_win - p_deal * v_deal - p_tsumo * v_tsumo
                + p_cont * V[t + 1])
        fold = fold_value(t, turns, p_tsumo, v_tsumo)
        if push >= fold:
            V[t], policy[t] = push, "push"
        else:
            V[t], policy[t] = fold, "fold"
    return V, policy

def first_push_turn(policy):
    """Turns are indexed 0..turns-1 from now; report the first turn at which
    pushing is optimal, or None."""
    for i, a in enumerate(policy):
        if a == "push":
            return i + 1
    return None

if __name__ == "__main__":
    TURNS = 12
    VW, VD, VT = 3900, 5200, 2600
    PT = 0.03
    print("One opponent visibly ready; {} turns remain.".format(TURNS))
    print("Own win +3900, dealing in -5200,")
    print(
        "opponent self-draw -2600 at {:.0%} per turn. Folding is absorbing.\n"
        .format(PT)
    )

    print(
        "A. Own hand ready, two-sided wait (p_win 0.10 per turn), deal-in"
        " risk 0.05:"
    )
    V, pol = solve(TURNS, 0.10, VW, 0.05, VD, PT, VT)
    print("   turn from now   policy   value from here   fold value here")
    for t in range(TURNS):
        print("   {:13d}   {:6s} {:16.1f} {:17.1f}"
              .format(t + 1, pol[t], V[t], fold_value(t, TURNS, PT, VT)))

    print(
        "\nB. Policy against own win probability, deal-in risk fixed at 0.05:"
    )
    print("   p_win   turns pushed (of {})   value now".format(TURNS))
    for pw in (0.02, 0.04, 0.06, 0.08, 0.12, 0.16):
        V, pol = solve(TURNS, pw, VW, 0.05, VD, PT, VT)
        print("   {:5.2f} {:22d} {:12.1f}"
              .format(pw, sum(1 for a in pol if a == "push"), V[0]))

    print(
        "\nC. Policy against deal-in risk, own win probability fixed at 0.08:"
    )
    print("   p_deal   turns pushed (of {})   value now".format(TURNS))
    for pd in (0.01, 0.03, 0.05, 0.08, 0.12):
        V, pol = solve(TURNS, 0.08, VW, pd, VD, PT, VT)
        print("   {:6.2f} {:22d} {:12.1f}"
              .format(pd, sum(1 for a in pol if a == "push"), V[0]))

    print(
        "\nD. The one-turn indifference condition. Ignoring continuation,"
        " pushing"
    )
    print("beats folding when p_win x {} >= p_deal x {},".format(VW, VD))
    print("i.e. p_win >= p_deal x {:.3f}.".format(VD / VW))
    print(
        "   (the opponent's self-draw term cancels: folding does not"
        " prevent it)"
    )
    print("   p_deal   break-even p_win")
    for pd in (0.01, 0.03, 0.05, 0.08, 0.12):
        print("   {:6.2f} {:18.3f}".format(pd, pd * VD / VW))
    print(
        "\n   The self-draw term cancelling is the structural point, and it"
        " is often"
    )
    print(
        "   stated backwards. Folding buys protection ONLY against dealing"
        " in; it"
    )
    print(
        "   buys nothing against the opponent drawing the tile themselves."
        " So the"
    )
    print(
        "   case for folding rests entirely on p_deal, and a nearly safe tile"
        " should"
    )
    print("   be pushed even with a weak hand.")

    print(
        "\nE. What this framing gets WRONG. Three things, in order of"
        " seriousness."
    )
    print(
        "   1. It is not a decision problem at all. p_deal depends on"
        " what the"
    )
    print(
        "      opponent waits on, and a thinking opponent chooses that"
        " partly in"
    )
    print(
        "      anticipation of being read (p03). Treating p_deal as a"
        " constant"
    )
    print(
        "      assumes a non-adaptive opponent, and Chapter 6 shows what"
        " changes"
    )
    print("      when that assumption is dropped.")
    print(
        "   2. There are three opponents, not one, and their hands are"
        " dependent"
    )
    print(
        "      through the tiles nobody has drawn. A tile safe against one"
        " can be"
    )
    print(
        "      the winning tile for another, so 'fold' is not a single"
        " action."
    )
    print(
        "   3. Points are not utility. In the last hand of a match, with a"
        " deficit"
    )
    print(
        "      to make up, the correct policy can be to take a"
        " negative-expectation"
    )
    print(
        "      push, which this objective cannot express. Chapter 6"
        " treats the"
    )
    print(
        "      bankroll version of that problem and Chapter 8 the match"
        " version."
    )
# One opponent visibly ready; 12 turns remain.
# Own win +3900, dealing in -5200,
# opponent self-draw -2600 at 3% per turn. Folding is absorbing.
#
# A. Own hand ready, two-sided wait (p_win 0.10 per turn), deal-in risk 0.05:
#    turn from now   policy   value from here   fold value here
#                1   push              262.2            -796.0
#                2   push              256.3            -740.2
#                3   push              249.2            -682.7
#                4   push              240.5            -623.4
#                5   push              229.8            -562.3
#                6   push              216.9            -499.2
#                7   push              201.1            -434.3
#                8   push              181.8            -367.3
#                9   push              158.3            -298.2
#               10   push              129.6            -227.1
#               11   push               94.6            -153.7
#               12   push               52.0             -78.0
#
# B. Policy against own win probability, deal-in risk fixed at 0.05:
#    p_win   turns pushed (of 12)   value now
#     0.02                      0       -796.0
#     0.04                      0       -796.0
#     0.06                      8       -609.8
#     0.08                     12       -142.4
#     0.12                     12        605.3
#     0.16                     12       1147.4
#
# C. Policy against deal-in risk, own win probability fixed at 0.08:
#    p_deal   turns pushed (of 12)   value now
#      0.01                     12       1189.6
#      0.03                     12        466.0
#      0.05                     12       -142.4
#      0.08                      2       -777.3
#      0.12                      0       -796.0
#
# D. The one-turn indifference condition. Ignoring continuation, pushing
# beats folding when p_win x 3900 >= p_deal x 5200,
# i.e. p_win >= p_deal x 1.333.
#    (the opponent's self-draw term cancels: folding does not prevent it)
#    p_deal   break-even p_win
#      0.01              0.013
#      0.03              0.040
#      0.05              0.067
#      0.08              0.107
#      0.12              0.160
#
#    The self-draw term cancelling is the structural point, and it is often
#    stated backwards. Folding buys protection ONLY against dealing in; it
#    buys nothing against the opponent drawing the tile themselves. So the
#    case for folding rests entirely on p_deal, and a nearly safe tile should
#    be pushed even with a weak hand.
#
# E. What this framing gets WRONG. Three things, in order of seriousness.
#    1. It is not a decision problem at all. p_deal depends on what the
#       opponent waits on, and a thinking opponent chooses that partly in
#       anticipation of being read (p03). Treating p_deal as a constant
#       assumes a non-adaptive opponent, and Chapter 6 shows what changes
#       when that assumption is dropped.
#    2. There are three opponents, not one, and their hands are dependent
#       through the tiles nobody has drawn. A tile safe against one can be
#       the winning tile for another, so 'fold' is not a single action.
#    3. Points are not utility. In the last hand of a match, with a deficit
#       to make up, the correct policy can be to take a negative-expectation
#       push, which this objective cannot express. Chapter 6 treats the
#       bankroll version of that problem and Chapter 8 the match version.

The term that cancels

Section D is the chapter’s main result, and it is often stated backwards.

The one-turn indifference condition is that pushing beats folding exactly when

pwin×3900≥pdeal×5200,p_{\text{win}}\times 3900 \ \ge\ p_{\text{deal}}\times 5200,

that is, pwin≥1.333pdealp_{\text{win}} \ge 1.333\,p_{\text{deal}}. The opponent’s self-draw term does not appear in the condition — because folding does not prevent a self-draw, the term is equal on both sides and cancels. Hence

folding protects only against dealing in. The case for folding therefore rests entirely on the deal-in probability, and a nearly safe tile should be pushed even with a weak hand.

The common formulation is “when in doubt, fold”, whereas by the condition above, what to look at when in doubt is how dangerous this tile is, not how weak the hand is. The two give opposite advice when the hand is weak and the tile is safe, which is a common situation.

The push-fold indifference boundary. The horizontal axis is the deal-in probability and the vertical axis one’s own win probability; the boundary is a line of slope 1.333 and is independent of the opponent’s self-draw probability. Push above the line, fold below it.

Sections B and C give the policy against the two parameters, and the direction is monotone: the higher one’s own win probability the longer one pushes, and the higher the deal-in probability the shorter. Section A gives a concrete trajectory with the fold value listed beside each turn, so that the comparison can be seen being made.

Three places where this framing fails

Section E states three, in order of seriousness.

First, it is not a decision problem at all. The deal-in probability depends on what the opponent waits on, and a thinking opponent chooses their wait and their discards partly in anticipation of being read (Chapter 3). Treating pdealp_{\text{deal}} as a constant assumes a non-adaptive opponent, and Chapter 6 shows that against an adaptive one no fixed rule is safe.

Second, there are three opponents, not one. Their hands are dependent through the tiles nobody drew, so a tile safe against one may be exactly another’s winning tile. “Folding” is therefore not a single action — folding against one can feed another. This book’s model handles one opponent, and that is an explicit simplification.

Third, points are not utility. In the last hand of a match, with a deficit to make up, the correct policy can be to accept a negative-expectation push, because the objective is placement rather than points. This objective function cannot express that. Chapter 6 handles the same problem at the level of the bankroll (Kelly and ruin), and Chapter 8 treats it as the terminal condition it is.

Hypothesis · not established

The author’s hypothesis (not established): against a fixed opponent pool, a policy that decides push or fold by this chapter’s indifference condition — looking only at the deal-in probability and one’s own win probability, and not at how “weak” the hand feels — beats a policy that decides primarily on hand strength. What would show it to be wrong: in the duplicate design of Chapter 8, a 95 % confidence interval for that comparison that includes zero or runs the other way.

Note

Relation to Volume 42. Volume 42 covered Bayesian search and optimal stopping, where the object is a single-agent decision problem and the stopping rule has clean optimality results. The difference here is precisely the first limitation above: the “environment” responds, so the standard results of optimal stopping apply only after the opponent has been frozen. Transferring single-agent stopping theory directly to a game is the commonest overreach in this kind of analysis.

Summary

Writing folding as a finite-horizon Markov decision process and solving it by backward induction gives a threshold policy: the higher one’s own win probability the longer one pushes, and the higher the deal-in probability the shorter. The main result is a structural conclusion that is often stated backwards: the opponent’s self-draw term cancels between pushing and folding, because folding protects only against dealing in, so the indifference condition contains only the deal-in probability and one’s own win probability (here pwin≥1.333pdealp_{\text{win}} \ge 1.333\,p_{\text{deal}}). A nearly safe tile should therefore be pushed even with a weak hand, which is the opposite of “when in doubt, fold”. The framing fails in three places: the deal-in probability depends on an opponent who adapts, so this is not a decision problem; there are three opponents with dependent hands, so “fold” is not a single action; and points are not utility, so in the last hand of a match the correct policy can be a negative-expectation push.

References

Bellman, Richard. 1957. Dynamic Programming. Princeton: Princeton University Press.
Chow, Y. S., Herbert Robbins, and David Siegmund. 1971. Great Expectations: The Theory of Optimal Stopping. Boston: Houghton Mifflin.
Puterman, Martin L. 1994. Markov Decision Processes: Discrete Stochastic Dynamic Programming. New York: Wiley.

Fan, Y. (2026). Imperfect-Information Stochastic Games, Bayesian Hand-Reading and Counterfactual Regret Minimisation in Four-Player Tile Dynamics: From Markov Transitions of the Tile Pool and the Optimal Stopping of Defensive Folding to Kelly Allocation in a Multiplayer Game, with a Textual Examination of the Orthodox and the Unorthodox in the Classical Military Writers [四人不完全信息随机博弈、贝叶斯读牌与反事实遗憾最小化] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24020464 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).