5D Library › Subscription Unit Economics, Net Revenue Retention and AI Inference Scaling Laws · 中文

Chapter 4. What a Token Costs: Prefill, Decode and the Memory Wall

Why AI software has a different cost structure

Traditional software has a marginal cost close to zero: serving one more user costs a little bandwidth and storage. Software built on a large language model has a marginal cost per use: every answer is computed on accelerators, and its cost scales with how much the user reads and writes. Gross margin, which for traditional subscription software is high and stable, becomes a function of usage, model size, batching and context length. This chapter computes that function from first principles; Chapter 5 shows what it does to a flat subscription price.

Evidence

The arithmetic of a forward pass. A dense transformer with NN parameters performs about 2N2N floating-point operations per token in a forward pass (Kaplan et al. 2020); the scaling-law literature that set model sizes also implies that serving cost grows with them (Hoffmann et al. 2022). Serving has two phases with different bottlenecks. Reading the prompt (prefill) processes many tokens in parallel and is limited by arithmetic. Writing the answer (decode) produces one token per step for each sequence in a batch, and at small batch sizes each step is limited by the memory bandwidth needed to read all the weights, plus each sequence’s attention key-value (KV) cache (Pope et al. 2023). Grouped-query attention shrinks the KV cache by sharing key and value heads (Ainslie et al. 2023), and paged memory management lets more sequences share the accelerator’s memory (Kwon et al. 2023).

The cost of a million tokens

The eighth program uses an illustrative accelerator — 101510^{15} FLOP/s, 3×10123\times10^{12} bytes/s of bandwidth, 80 GB of memory, 2.00 dollars an hour — and two illustrative dense model sizes. The numbers are not quotes for any product; they show how the costs scale.

Calculation

# Chapter 4. What a token costs to serve. An illustrative accelerator: 1.0e15
# dense FLOP/s, 3.0e12 bytes/s of memory bandwidth, 80e9 bytes of memory, 2.00
# dollars an hour. Two illustrative dense transformer sizes with grouped-query
# attention, weights in 16-bit (2 bytes per parameter):
#   "8B":  8e9 parameters, 32 layers, 8 KV heads of dimension 128, on 1 device
#   "70B": 70e9 parameters, 80 layers, 8 KV heads of dimension 128, on 4
#   devices
# KV cache per token = 2 (K and V) x layers x KV heads x head dim x 2 bytes.
# PREFILL (reading the prompt) is compute-bound: about 2 x params FLOPs per
# token, at 50 % utilisation. DECODE (writing the answer) runs one step per
# output token for the whole batch; each step must read all weights plus every
# sequence's KV cache from memory, and also do 2 x params FLOPs per sequence;
# the step takes whichever is longer.
def model(name):
    if name == "8B":
        return dict(P=8e9, L=32, H=8, Dh=128, dev=1)
    return dict(P=70e9, L=80, H=8, Dh=128, dev=4)

FLOPS, BW, MEM, PRICE = 1.0e15, 3.0e12, 80e9, 2.00

def kv_bytes(m):
    return 2 * m["L"] * m["H"] * m["Dh"] * 2

def prefill_cost_per_m(m):
    sec = 2 * m["P"] * 1e6 / (FLOPS * m["dev"] * 0.5)
    return sec * PRICE * m["dev"] / 3600

def decode(m, batch, ctx):
    w = 2 * m["P"]
    kv = batch * ctx * kv_bytes(m)
    if w + kv > MEM * m["dev"]:
        return None
    t_mem = (w + kv) / (BW * m["dev"])
    t_flop = 2 * m["P"] * batch / (FLOPS * m["dev"])
    t = max(t_mem, t_flop)
    return (
        t / batch * 1e6 * PRICE * m["dev"] / 3600,
        (w + kv) / (MEM * m["dev"]),
        kv / (w + kv),
    )

if __name__ == "__main__":
    for name in ("8B", "70B"):
        m = model(name)
        print("{}: KV cache {:,} bytes per token; prefill {:.3f} dollars per"
              .format(name, kv_bytes(m), prefill_cost_per_m(m)))
        print("   million input tokens")
        print(
            "   batch   context   decode $/M      memory used   KV share of"
        )
        print("                     output tokens                 bytes read")
        for ctx in (2000, 32000):
            for batch in (1, 8, 32, 128):
                r = decode(m, batch, ctx)
                if r is None:
                    print(
                        "   {:>5}   {:>6}    does not fit in memory".format(
                            batch, ctx
                        )
                    )
                else:
                    print(
                        "   {:>5}   {:>6}    {:9.3f}          {:6.1%}       "
                        " {:6.1%}".format(batch, ctx, r[0], r[1], r[2])
                    )
        print()
    print(
        "Reading: output tokens cost far more than input tokens at small"
        " batches,"
    )
    print(
        "because every decode step re-reads the weights. Batching spreads"
        " that read"
    )
    print(
        "over many users until the KV cache - which grows with context -"
        " takes over"
    )
    print(
        "the memory and the bandwidth. Long contexts cap the batch, and so"
        " the price."
    )
# 8B: KV cache 131,072 bytes per token; prefill 0.018 dollars per
#    million input tokens
#    batch   context   decode $/M      memory used   KV share of
#                      output tokens                 bytes read
#        1     2000        3.012           20.3%          1.6%
#        8     2000        0.419           22.6%         11.6%
#       32     2000        0.141           30.5%         34.4%
#      128     2000        0.072           61.9%         67.7%
#        1    32000        3.740           25.2%         20.8%
#        8    32000        1.147           61.9%         67.7%
#       32    32000    does not fit in memory
#      128    32000    does not fit in memory
#
# 70B: KV cache 327,680 bytes per token; prefill 0.156 dollars per
#    million input tokens
#    batch   context   decode $/M      memory used   KV share of
#                      output tokens                 bytes read
#        1     2000       26.047           44.0%          0.5%
#        8     2000        3.362           45.4%          3.6%
#       32     2000        0.932           50.3%         13.0%
#      128     2000        0.324           70.0%         37.5%
#        1    32000       27.868           47.0%          7.0%
#        8    32000        5.183           70.0%         37.5%
#       32    32000    does not fit in memory
#      128    32000    does not fit in memory
#
# Reading: output tokens cost far more than input tokens at small batches,
# because every decode step re-reads the weights. Batching spreads that read
# over many users until the KV cache - which grows with context - takes over
# the memory and the bandwidth. Long contexts cap the batch, and so the price.

Three facts follow.

Output costs far more than input at small batches. For the 8B model, a million input tokens cost 0.018 dollars to prefill, while a million output tokens at batch 1 cost 3.012: every decode step re-reads all the weights to produce one token per sequence.

Batching is where the margin is. At 2,000 tokens of context, raising the batch from 1 to 128 cuts the 8B model’s output cost from 3.012 to 0.072 dollars per million tokens, and the 70B model’s from 26.047 to 0.324. The weights are read once per step and shared by every sequence in the batch.

Context length caps the batch. The KV cache grows with every token of context: 131,072 bytes per token for the 8B model, 327,680 for the 70B. At 32,000 tokens of context the 8B model cannot fit a batch of 32 in memory, and at batch 8 its output cost is 1.147 against 0.419 at short context; the 70B model fits only 8 sequences. Long context is not merely more input to read; it removes the batching that made output cheap.

Decode cost per million output tokens against batch size for the two illustrative models at 2,000 and 32,000 tokens of context. Lines stop where the batch no longer fits in memory.

Agents re-read themselves

The ninth program applies those unit prices to multi-turn work. A conversation or agent loop re-sends the whole history as input on every turn.

Calculation

# Chapter 4. Why multi-turn and agent workloads cost more than their output
# suggests. Unit prices are taken from p08's 70B rows at batch 32 and 2,000
# tokens of context: 0.156 dollars per million input (prefill) tokens and
# 0.932 per million output tokens. A conversation starts with a 2,000-token
# system prompt and tools description; each turn adds 300 tokens from the user
# (or tool results) and 400 generated tokens, and the whole conversation so
# far is sent again as input on the next turn. With prefix caching only the
# tokens not already processed are prefilled; a cache hit rate h means a share
# h of the previously seen prefix is found in cache (the rest is prefilled
# again).
IN, OUT = 0.156e-6, 0.932e-6
SYS, USER, GEN = 2000, 300, 400

def cost(turns, h):
    ctx, c_in, c_out, toks_in = SYS, 0.0, 0.0, 0
    for k in range(turns):
        new = (SYS if k == 0 else 0) + USER
        seen = ctx - (SYS if k == 0 else 0)
        prefill = new + (1 - h) * (seen if k > 0 else 0)
        toks_in += ctx + USER if k > 0 else SYS + USER
        c_in += prefill * IN
        c_out += GEN * OUT
        ctx += USER + GEN
    return c_in, c_out, toks_in

if __name__ == "__main__":
    print(
        "   turns   input tokens   cost:       cost:        cost:       "
        " output share"
    )
    print(
        "           sent           no cache    50 % hits    95 % hits    (no"
        " cache)"
    )
    for turns in (1, 5, 20, 50, 100):
        ci0, co, tin = cost(turns, 0.0)
        ci5, _, _ = cost(turns, 0.5)
        ci9, _, _ = cost(turns, 0.95)
        print(
            "   {:>5}   {:>12,}   {:9.4f}   {:9.4f}    {:9.4f}       {:6.1%}"
            .format(turns, tin, ci0 + co, ci5 + co, ci9 + co, co / (ci0 + co))
        )
    print(
        "\nReading: re-sending a growing context makes input tokens grow"
        " with the"
    )
    print(
        "square of the number of turns while output grows linearly; without"
        " caching,"
    )
    print(
        "a long agent loop is mostly paying to re-read itself. Caching"
        " restores"
    )
    print("roughly linear cost.")
#    turns   input tokens   cost:       cost:        cost:        output share
#            sent           no cache    50 % hits    95 % hits    (no cache)
#        1          2,300      0.0007      0.0007       0.0007        51.0%
#        5         18,500      0.0047      0.0036       0.0025        39.2%
#       20        179,000      0.0354      0.0220       0.0100        21.1%
#       50        972,500      0.1704      0.0958       0.0287        10.9%
#      100      3,695,000      0.6137      0.3280       0.0708         6.1%
#
# Reading: re-sending a growing context makes input tokens grow with the
# square of the number of turns while output grows linearly; without caching,
# a long agent loop is mostly paying to re-read itself. Caching restores
# roughly linear cost.

Input tokens grow with the square of the number of turns: one turn sends 2,300 tokens, twenty turns 179,000 and a hundred turns 3,695,000, while output grows linearly. Without caching, output is 51.0 % of the cost of a one-turn exchange but only 6.1 % of a hundred-turn one: a long agent loop is mostly paying to re-read itself. Prefix caching restores roughly linear cost — at 95 % cache hits the hundred-turn conversation costs 0.0708 instead of 0.6137.

Note

What the program leaves out. It prices every token of a long conversation at the short-context rate. By the eighth program, long contexts also cut the feasible batch and raise the output price, so the real cost of a long agent loop without careful context management is higher than shown.

Summary

For a model-based product, marginal cost is per use. Prefill is cheap and compute-bound; decode is memory-bound and expensive at small batches because every step re-reads the weights. Batching spreads that cost — a factor of about forty for the 8B model from batch 1 to 128 — until the KV cache, which grows with context, fills memory and caps the batch. Multi-turn and agent workloads re-send their history, so input grows with the square of the turns; caching is not an optimisation but the difference between linear and quadratic cost.

References

Ainslie, Joshua, James Lee-Thorp, et al. 2023. “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.” Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.
Hoffmann, Jordan, Sebastian Borgeaud, et al. 2022. Training Compute-Optimal Large Language Models. arXiv:2203.15556.
Kaplan, Jared, Sam McCandlish, et al. 2020. Scaling Laws for Neural Language Models. arXiv:2001.08361.
Kwon, Woosuk, Zhuohan Li, et al. 2023. “Efficient Memory Management for Large Language Model Serving with PagedAttention.” Proceedings of the 29th Symposium on Operating Systems Principles.
Pope, Reiner, Sholto Douglas, et al. 2023. “Efficiently Scaling Transformer Inference.” Proceedings of Machine Learning and Systems 5.

Fan, Y. (2026). Subscription Unit Economics, Net Revenue Retention and AI Inference Scaling Laws: Cohort Survival Mixtures, Negative-Churn Differential Equations, Token Compute Margins and Metcalfe Congestion Limits [订阅制单位经济学、净收入留存与人工智能推理标度律] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090164 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).