What does Subscription and AI Unit Economics conclude, in numbers, about “What a Token Costs: Prefill, Decode and the Memory Wall”?

Answer: Chapter 4 computes token costs from first principles: at batch 1 the 8B model's output costs 3.012 dollars per million against 0.018 for input; batching cuts it to 0.072; long context removes the batch; agent loops re-send their history, so output is only 6.1 % of the cost of a hundred-turn conversation, and caching cuts 0.6137 to 0.0708.

Scope of the numbers: It is not investment or financial advice; all data are synthetic, generated by its programs. Chapter 7 reads the Shiji's "the greedy merchant makes three, the modest merchant five" and Jiran's "let no money lie idle", marking the margin-against-turnover reading as an interpretation.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. printed values that match the answer: 0.018, 0.072, 3.012

8B: KV cache 131,072 bytes per token; prefill 0.018 dollars per
million input tokens
batch context decode $/M memory used KV share of
output tokens bytes read
12000 3.012 20.3% 1.6%
82000 0.419 22.6% 11.6%
322000 0.141 30.5% 34.4%
1282000 0.072 61.9% 67.7%
132000 3.740 25.2% 20.8%
832000 1.147 61.9% 67.7%
3232000 does not fit in memory
12832000 does not fit in memory
70B: KV cache 327,680 bytes per token; prefill 0.156 dollars per
million input tokens
batch context decode $/M memory used KV share of
output tokens bytes read
12000 26.047 44.0% 0.5%
82000 3.362 45.4% 3.6%
322000 0.932 50.3% 13.0%
1282000 0.324 70.0% 37.5%
132000 27.868 47.0% 7.0%
832000 5.183 70.0% 37.5%
3232000 does not fit in memory
12832000 does not fit in memory
Reading: output tokens cost far more than input tokens at small batches,
because every decode step re-reads the weights. Batching spreads that read
over many users until the KV cache - which grows with context - takes over
the memory and the bandwidth. Long contexts cap the batch, and so the price.
Show the full program
# Chapter 4. What a token costs to serve. An illustrative accelerator: 1.0e15
# dense FLOP/s, 3.0e12 bytes/s of memory bandwidth, 80e9 bytes of memory, 2.00
# dollars an hour. Two illustrative dense transformer sizes with grouped-query
# attention, weights in 16-bit (2 bytes per parameter):
#   "8B":  8e9 parameters, 32 layers, 8 KV heads of dimension 128, on 1 device
#   "70B": 70e9 parameters, 80 layers, 8 KV heads of dimension 128, on 4
#   devices
# KV cache per token = 2 (K and V) x layers x KV heads x head dim x 2 bytes.
# PREFILL (reading the prompt) is compute-bound: about 2 x params FLOPs per
# token, at 50 % utilisation. DECODE (writing the answer) runs one step per
# output token for the whole batch; each step must read all weights plus every
# sequence's KV cache from memory, and also do 2 x params FLOPs per sequence;
# the step takes whichever is longer.
def model(name):
    if name == "8B":
        return dict(P=8e9, L=32, H=8, Dh=128, dev=1)
    return dict(P=70e9, L=80, H=8, Dh=128, dev=4)

FLOPS, BW, MEM, PRICE = 1.0e15, 3.0e12, 80e9, 2.00

def kv_bytes(m):
    return 2 * m["L"] * m["H"] * m["Dh"] * 2

def prefill_cost_per_m(m):
    sec = 2 * m["P"] * 1e6 / (FLOPS * m["dev"] * 0.5)
    return sec * PRICE * m["dev"] / 3600

def decode(m, batch, ctx):
    w = 2 * m["P"]
    kv = batch * ctx * kv_bytes(m)
    if w + kv > MEM * m["dev"]:
        return None
    t_mem = (w + kv) / (BW * m["dev"])
    t_flop = 2 * m["P"] * batch / (FLOPS * m["dev"])
    t = max(t_mem, t_flop)
    return (
        t / batch * 1e6 * PRICE * m["dev"] / 3600,
        (w + kv) / (MEM * m["dev"]),
        kv / (w + kv),
    )

if __name__ == "__main__":
    for name in ("8B", "70B"):
        m = model(name)
        print("{}: KV cache {:,} bytes per token; prefill {:.3f} dollars per"
              .format(name, kv_bytes(m), prefill_cost_per_m(m)))
        print("   million input tokens")
        print(
            "   batch   context   decode $/M      memory used   KV share of"
        )
        print("                     output tokens                 bytes read")
        for ctx in (2000, 32000):
            for batch in (1, 8, 32, 128):
                r = decode(m, batch, ctx)
                if r is None:
                    print(
                        "   {:>5}   {:>6}    does not fit in memory".format(
                            batch, ctx
                        )
                    )
                else:
                    print(
                        "   {:>5}   {:>6}    {:9.3f}          {:6.1%}       "
                        " {:6.1%}".format(batch, ctx, r[0], r[1], r[2])
                    )
        print()
    print(
        "Reading: output tokens cost far more than input tokens at small"
        " batches,"
    )
    print(
        "because every decode step re-reads the weights. Batching spreads"
        " that read"
    )
    print(
        "over many users until the KV cache - which grows with context -"
        " takes over"
    )
    print(
        "the memory and the bandwidth. Long contexts cap the batch, and so"
        " the price."
    )

Single source and citation

Fan, Y. (2026). Subscription Unit Economics, Net Revenue Retention and AI Inference Scaling Laws (Version 1.0.0), Chapter 4: What a Token Costs: Prefill, Decode and the Memory Wall. 5D Library. https://doi.org/10.5281/zenodo.24090164

DOI: 10.5281/zenodo.24090164 · Read the monograph · 5DL-B-058 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026058,
  author    = {Fan, Yafu},
  title     = {Subscription Unit Economics, Net Revenue Retention and AI Inference Scaling Laws},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  doi       = {10.5281/zenodo.24090164},
  url       = {https://doi.org/10.5281/zenodo.24090164}
}

Other chapter answers in this book