5D Library › Subscription Unit Economics, Net Revenue Retention and AI Inference Scaling Laws · 中文
Chapter 4. What a Token Costs: Prefill, Decode and the Memory Wall
Why AI software has a different cost structure
Traditional software has a marginal cost close to zero: serving one more user costs a little bandwidth and storage. Software built on a large language model has a marginal cost per use: every answer is computed on accelerators, and its cost scales with how much the user reads and writes. Gross margin, which for traditional subscription software is high and stable, becomes a function of usage, model size, batching and context length. This chapter computes that function from first principles; Chapter 5 shows what it does to a flat subscription price.
Evidence
The arithmetic of a forward pass. A dense transformer with parameters performs about floating-point operations per token in a forward pass (Kaplan et al. 2020); the scaling-law literature that set model sizes also implies that serving cost grows with them (Hoffmann et al. 2022). Serving has two phases with different bottlenecks. Reading the prompt (prefill) processes many tokens in parallel and is limited by arithmetic. Writing the answer (decode) produces one token per step for each sequence in a batch, and at small batch sizes each step is limited by the memory bandwidth needed to read all the weights, plus each sequence’s attention key-value (KV) cache (Pope et al. 2023). Grouped-query attention shrinks the KV cache by sharing key and value heads (Ainslie et al. 2023), and paged memory management lets more sequences share the accelerator’s memory (Kwon et al. 2023).
The cost of a million tokens
The eighth program uses an illustrative accelerator — FLOP/s, bytes/s of bandwidth, 80 GB of memory, 2.00 dollars an hour — and two illustrative dense model sizes. The numbers are not quotes for any product; they show how the costs scale.
Calculation
# Chapter 4. What a token costs to serve. An illustrative accelerator: 1.0e15
# dense FLOP/s, 3.0e12 bytes/s of memory bandwidth, 80e9 bytes of memory, 2.00
# dollars an hour. Two illustrative dense transformer sizes with grouped-query
# attention, weights in 16-bit (2 bytes per parameter):
# "8B": 8e9 parameters, 32 layers, 8 KV heads of dimension 128, on 1 device
# "70B": 70e9 parameters, 80 layers, 8 KV heads of dimension 128, on 4
# devices
# KV cache per token = 2 (K and V) x layers x KV heads x head dim x 2 bytes.
# PREFILL (reading the prompt) is compute-bound: about 2 x params FLOPs per
# token, at 50 % utilisation. DECODE (writing the answer) runs one step per
# output token for the whole batch; each step must read all weights plus every
# sequence's KV cache from memory, and also do 2 x params FLOPs per sequence;
# the step takes whichever is longer.
def model(name):
if name == "8B":
return dict(P=8e9, L=32, H=8, Dh=128, dev=1)
return dict(P=70e9, L=80, H=8, Dh=128, dev=4)
FLOPS, BW, MEM, PRICE = 1.0e15, 3.0e12, 80e9, 2.00
def kv_bytes(m):
return 2 * m["L"] * m["H"] * m["Dh"] * 2
def prefill_cost_per_m(m):
sec = 2 * m["P"] * 1e6 / (FLOPS * m["dev"] * 0.5)
return sec * PRICE * m["dev"] / 3600
def decode(m, batch, ctx):
w = 2 * m["P"]
kv = batch * ctx * kv_bytes(m)
if w + kv > MEM * m["dev"]:
return None
t_mem = (w + kv) / (BW * m["dev"])
t_flop = 2 * m["P"] * batch / (FLOPS * m["dev"])
t = max(t_mem, t_flop)
return (
t / batch * 1e6 * PRICE * m["dev"] / 3600,
(w + kv) / (MEM * m["dev"]),
kv / (w + kv),
)
if __name__ == "__main__":
for name in ("8B", "70B"):
m = model(name)
print("{}: KV cache {:,} bytes per token; prefill {:.3f} dollars per"
.format(name, kv_bytes(m), prefill_cost_per_m(m)))
print(" million input tokens")
print(
" batch context decode $/M memory used KV share of"
)
print(" output tokens bytes read")
for ctx in (2000, 32000):
for batch in (1, 8, 32, 128):
r = decode(m, batch, ctx)
if r is None:
print(
" {:>5} {:>6} does not fit in memory".format(
batch, ctx
)
)
else:
print(
" {:>5} {:>6} {:9.3f} {:6.1%} "
" {:6.1%}".format(batch, ctx, r[0], r[1], r[2])
)
print()
print(
"Reading: output tokens cost far more than input tokens at small"
" batches,"
)
print(
"because every decode step re-reads the weights. Batching spreads"
" that read"
)
print(
"over many users until the KV cache - which grows with context -"
" takes over"
)
print(
"the memory and the bandwidth. Long contexts cap the batch, and so"
" the price."
)
# 8B: KV cache 131,072 bytes per token; prefill 0.018 dollars per
# million input tokens
# batch context decode $/M memory used KV share of
# output tokens bytes read
# 1 2000 3.012 20.3% 1.6%
# 8 2000 0.419 22.6% 11.6%
# 32 2000 0.141 30.5% 34.4%
# 128 2000 0.072 61.9% 67.7%
# 1 32000 3.740 25.2% 20.8%
# 8 32000 1.147 61.9% 67.7%
# 32 32000 does not fit in memory
# 128 32000 does not fit in memory
#
# 70B: KV cache 327,680 bytes per token; prefill 0.156 dollars per
# million input tokens
# batch context decode $/M memory used KV share of
# output tokens bytes read
# 1 2000 26.047 44.0% 0.5%
# 8 2000 3.362 45.4% 3.6%
# 32 2000 0.932 50.3% 13.0%
# 128 2000 0.324 70.0% 37.5%
# 1 32000 27.868 47.0% 7.0%
# 8 32000 5.183 70.0% 37.5%
# 32 32000 does not fit in memory
# 128 32000 does not fit in memory
#
# Reading: output tokens cost far more than input tokens at small batches,
# because every decode step re-reads the weights. Batching spreads that read
# over many users until the KV cache - which grows with context - takes over
# the memory and the bandwidth. Long contexts cap the batch, and so the price.Three facts follow.
Output costs far more than input at small batches. For the 8B model, a million input tokens cost 0.018 dollars to prefill, while a million output tokens at batch 1 cost 3.012: every decode step re-reads all the weights to produce one token per sequence.
Batching is where the margin is. At 2,000 tokens of context, raising the batch from 1 to 128 cuts the 8B model’s output cost from 3.012 to 0.072 dollars per million tokens, and the 70B model’s from 26.047 to 0.324. The weights are read once per step and shared by every sequence in the batch.
Context length caps the batch. The KV cache grows with every token of context: 131,072 bytes per token for the 8B model, 327,680 for the 70B. At 32,000 tokens of context the 8B model cannot fit a batch of 32 in memory, and at batch 8 its output cost is 1.147 against 0.419 at short context; the 70B model fits only 8 sequences. Long context is not merely more input to read; it removes the batching that made output cheap.
Agents re-read themselves
The ninth program applies those unit prices to multi-turn work. A conversation or agent loop re-sends the whole history as input on every turn.
Calculation
# Chapter 4. Why multi-turn and agent workloads cost more than their output
# suggests. Unit prices are taken from p08's 70B rows at batch 32 and 2,000
# tokens of context: 0.156 dollars per million input (prefill) tokens and
# 0.932 per million output tokens. A conversation starts with a 2,000-token
# system prompt and tools description; each turn adds 300 tokens from the user
# (or tool results) and 400 generated tokens, and the whole conversation so
# far is sent again as input on the next turn. With prefix caching only the
# tokens not already processed are prefilled; a cache hit rate h means a share
# h of the previously seen prefix is found in cache (the rest is prefilled
# again).
IN, OUT = 0.156e-6, 0.932e-6
SYS, USER, GEN = 2000, 300, 400
def cost(turns, h):
ctx, c_in, c_out, toks_in = SYS, 0.0, 0.0, 0
for k in range(turns):
new = (SYS if k == 0 else 0) + USER
seen = ctx - (SYS if k == 0 else 0)
prefill = new + (1 - h) * (seen if k > 0 else 0)
toks_in += ctx + USER if k > 0 else SYS + USER
c_in += prefill * IN
c_out += GEN * OUT
ctx += USER + GEN
return c_in, c_out, toks_in
if __name__ == "__main__":
print(
" turns input tokens cost: cost: cost: "
" output share"
)
print(
" sent no cache 50 % hits 95 % hits (no"
" cache)"
)
for turns in (1, 5, 20, 50, 100):
ci0, co, tin = cost(turns, 0.0)
ci5, _, _ = cost(turns, 0.5)
ci9, _, _ = cost(turns, 0.95)
print(
" {:>5} {:>12,} {:9.4f} {:9.4f} {:9.4f} {:6.1%}"
.format(turns, tin, ci0 + co, ci5 + co, ci9 + co, co / (ci0 + co))
)
print(
"\nReading: re-sending a growing context makes input tokens grow"
" with the"
)
print(
"square of the number of turns while output grows linearly; without"
" caching,"
)
print(
"a long agent loop is mostly paying to re-read itself. Caching"
" restores"
)
print("roughly linear cost.")
# turns input tokens cost: cost: cost: output share
# sent no cache 50 % hits 95 % hits (no cache)
# 1 2,300 0.0007 0.0007 0.0007 51.0%
# 5 18,500 0.0047 0.0036 0.0025 39.2%
# 20 179,000 0.0354 0.0220 0.0100 21.1%
# 50 972,500 0.1704 0.0958 0.0287 10.9%
# 100 3,695,000 0.6137 0.3280 0.0708 6.1%
#
# Reading: re-sending a growing context makes input tokens grow with the
# square of the number of turns while output grows linearly; without caching,
# a long agent loop is mostly paying to re-read itself. Caching restores
# roughly linear cost.Input tokens grow with the square of the number of turns: one turn sends 2,300 tokens, twenty turns 179,000 and a hundred turns 3,695,000, while output grows linearly. Without caching, output is 51.0 % of the cost of a one-turn exchange but only 6.1 % of a hundred-turn one: a long agent loop is mostly paying to re-read itself. Prefix caching restores roughly linear cost — at 95 % cache hits the hundred-turn conversation costs 0.0708 instead of 0.6137.
Note
What the program leaves out. It prices every token of a long conversation at the short-context rate. By the eighth program, long contexts also cut the feasible batch and raise the output price, so the real cost of a long agent loop without careful context management is higher than shown.
Summary
For a model-based product, marginal cost is per use. Prefill is cheap and compute-bound; decode is memory-bound and expensive at small batches because every step re-reads the weights. Batching spreads that cost — a factor of about forty for the 8B model from batch 1 to 128 — until the KV cache, which grows with context, fills memory and caps the batch. Multi-turn and agent workloads re-send their history, so input grows with the square of the turns; caching is not an optimisation but the difference between linear and quadratic cost.
References
Fan, Y. (2026). Subscription Unit Economics, Net Revenue Retention and AI Inference Scaling Laws: Cohort Survival Mixtures, Negative-Churn Differential Equations, Token Compute Margins and Metcalfe Congestion Limits [订阅制单位经济学、净收入留存与人工智能推理标度律] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090164 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).