5D Library › 订阅制单位经济学、净收入留存与人工智能推理标度律 · English
第四章 一个 Token 的成本:预填充、解码与内存墙
为什么 AI 软件的成本结构不同
传统软件的边际成本接近于零:多服务一个用户,只多花一点带宽和存储。建立在大语言模型之上的软件,每一次使用都有边际成本:每一个回答都在加速器上计算,成本随用户读写的多少而变化。对传统订阅软件来说又高又稳定的毛利率,变成了用量、模型规模、批处理与上下文长度的函数。本章从第一性原理计算这个函数;第五章说明它对固定月费的订阅价格做了什么。
证据
一次前向计算的算术。 一个有 个参数的稠密 Transformer,每个 token 的前向计算约需 次浮点运算 (Kaplan et al. 2020);决定模型规模的标度律文献,也意味着服务成本随规模增长 (Hoffmann et al. 2022)。推理服务有两个瓶颈不同的阶段。读取提示词(预填充)并行处理大量 token,受算力限制。写出回答(解码)每一步为批中的每个序列产生一个 token,在小批量时每一步受内存带宽限制:需要读取全部权重,外加每个序列注意力的键-值(KV)缓存 (Pope et al. 2023)。分组查询注意力通过共享键与值的头来缩小 KV 缓存 (Ainslie et al. 2023),分页式内存管理让更多序列共享加速器的内存 (Kwon et al. 2023)。
一百万个 token 的成本
第八段程序使用一个示意性的加速器——每秒 次浮点运算、每秒 字节的带宽、80 GB 内存、每小时 2.00 美元——以及两种示意性的稠密模型规模。这些数字不是任何产品的报价;它们说明成本如何随规模变化。
推导
# Chapter 4. What a token costs to serve. An illustrative accelerator: 1.0e15
# dense FLOP/s, 3.0e12 bytes/s of memory bandwidth, 80e9 bytes of memory, 2.00
# dollars an hour. Two illustrative dense transformer sizes with grouped-query
# attention, weights in 16-bit (2 bytes per parameter):
# "8B": 8e9 parameters, 32 layers, 8 KV heads of dimension 128, on 1 device
# "70B": 70e9 parameters, 80 layers, 8 KV heads of dimension 128, on 4
# devices
# KV cache per token = 2 (K and V) x layers x KV heads x head dim x 2 bytes.
# PREFILL (reading the prompt) is compute-bound: about 2 x params FLOPs per
# token, at 50 % utilisation. DECODE (writing the answer) runs one step per
# output token for the whole batch; each step must read all weights plus every
# sequence's KV cache from memory, and also do 2 x params FLOPs per sequence;
# the step takes whichever is longer.
def model(name):
if name == "8B":
return dict(P=8e9, L=32, H=8, Dh=128, dev=1)
return dict(P=70e9, L=80, H=8, Dh=128, dev=4)
FLOPS, BW, MEM, PRICE = 1.0e15, 3.0e12, 80e9, 2.00
def kv_bytes(m):
return 2 * m["L"] * m["H"] * m["Dh"] * 2
def prefill_cost_per_m(m):
sec = 2 * m["P"] * 1e6 / (FLOPS * m["dev"] * 0.5)
return sec * PRICE * m["dev"] / 3600
def decode(m, batch, ctx):
w = 2 * m["P"]
kv = batch * ctx * kv_bytes(m)
if w + kv > MEM * m["dev"]:
return None
t_mem = (w + kv) / (BW * m["dev"])
t_flop = 2 * m["P"] * batch / (FLOPS * m["dev"])
t = max(t_mem, t_flop)
return (
t / batch * 1e6 * PRICE * m["dev"] / 3600,
(w + kv) / (MEM * m["dev"]),
kv / (w + kv),
)
if __name__ == "__main__":
for name in ("8B", "70B"):
m = model(name)
print("{}: KV cache {:,} bytes per token; prefill {:.3f} dollars per"
.format(name, kv_bytes(m), prefill_cost_per_m(m)))
print(" million input tokens")
print(
" batch context decode $/M memory used KV share of"
)
print(" output tokens bytes read")
for ctx in (2000, 32000):
for batch in (1, 8, 32, 128):
r = decode(m, batch, ctx)
if r is None:
print(
" {:>5} {:>6} does not fit in memory".format(
batch, ctx
)
)
else:
print(
" {:>5} {:>6} {:9.3f} {:6.1%} "
" {:6.1%}".format(batch, ctx, r[0], r[1], r[2])
)
print()
print(
"Reading: output tokens cost far more than input tokens at small"
" batches,"
)
print(
"because every decode step re-reads the weights. Batching spreads"
" that read"
)
print(
"over many users until the KV cache - which grows with context -"
" takes over"
)
print(
"the memory and the bandwidth. Long contexts cap the batch, and so"
" the price."
)
# 8B: KV cache 131,072 bytes per token; prefill 0.018 dollars per
# million input tokens
# batch context decode $/M memory used KV share of
# output tokens bytes read
# 1 2000 3.012 20.3% 1.6%
# 8 2000 0.419 22.6% 11.6%
# 32 2000 0.141 30.5% 34.4%
# 128 2000 0.072 61.9% 67.7%
# 1 32000 3.740 25.2% 20.8%
# 8 32000 1.147 61.9% 67.7%
# 32 32000 does not fit in memory
# 128 32000 does not fit in memory
#
# 70B: KV cache 327,680 bytes per token; prefill 0.156 dollars per
# million input tokens
# batch context decode $/M memory used KV share of
# output tokens bytes read
# 1 2000 26.047 44.0% 0.5%
# 8 2000 3.362 45.4% 3.6%
# 32 2000 0.932 50.3% 13.0%
# 128 2000 0.324 70.0% 37.5%
# 1 32000 27.868 47.0% 7.0%
# 8 32000 5.183 70.0% 37.5%
# 32 32000 does not fit in memory
# 128 32000 does not fit in memory
#
# Reading: output tokens cost far more than input tokens at small batches,
# because every decode step re-reads the weights. Batching spreads that read
# over many users until the KV cache - which grows with context - takes over
# the memory and the bandwidth. Long contexts cap the batch, and so the price.由此得出三个事实。
小批量时输出比输入贵得多。 对 8B 模型,预填充一百万个输入 token 花 0.018 美元,而批量为 1 时一百万个输出 token 花 3.012 美元:每个解码步都要重新读取全部权重,才为每个序列产生一个 token。
毛利在批处理里。 上下文为 2,000 个 token 时,把批量从 1 提到 128,8B 模型的输出成本从每百万 token 3.012 美元降到 0.072,70B 模型从 26.047 降到 0.324。权重每步只读一次,由批中的每个序列分摊。
上下文长度给批量封顶。 KV 缓存随上下文的每个 token 增长:8B 模型每个 token 131,072 字节,70B 模型 327,680 字节。上下文为 32,000 个 token 时,8B 模型装不下批量 32,批量为 8 时输出成本为 1.147,而短上下文时为 0.419;70B 模型只装得下 8 个序列。长上下文不只是要读的输入更多;它拿走了让输出变便宜的批处理。
智能体在反复重读自己
第九段程序把这些单价用于多轮工作。一段对话或一个智能体循环,每一轮都把全部历史作为输入重新发送。
推导
# Chapter 4. Why multi-turn and agent workloads cost more than their output
# suggests. Unit prices are taken from p08's 70B rows at batch 32 and 2,000
# tokens of context: 0.156 dollars per million input (prefill) tokens and
# 0.932 per million output tokens. A conversation starts with a 2,000-token
# system prompt and tools description; each turn adds 300 tokens from the user
# (or tool results) and 400 generated tokens, and the whole conversation so
# far is sent again as input on the next turn. With prefix caching only the
# tokens not already processed are prefilled; a cache hit rate h means a share
# h of the previously seen prefix is found in cache (the rest is prefilled
# again).
IN, OUT = 0.156e-6, 0.932e-6
SYS, USER, GEN = 2000, 300, 400
def cost(turns, h):
ctx, c_in, c_out, toks_in = SYS, 0.0, 0.0, 0
for k in range(turns):
new = (SYS if k == 0 else 0) + USER
seen = ctx - (SYS if k == 0 else 0)
prefill = new + (1 - h) * (seen if k > 0 else 0)
toks_in += ctx + USER if k > 0 else SYS + USER
c_in += prefill * IN
c_out += GEN * OUT
ctx += USER + GEN
return c_in, c_out, toks_in
if __name__ == "__main__":
print(
" turns input tokens cost: cost: cost: "
" output share"
)
print(
" sent no cache 50 % hits 95 % hits (no"
" cache)"
)
for turns in (1, 5, 20, 50, 100):
ci0, co, tin = cost(turns, 0.0)
ci5, _, _ = cost(turns, 0.5)
ci9, _, _ = cost(turns, 0.95)
print(
" {:>5} {:>12,} {:9.4f} {:9.4f} {:9.4f} {:6.1%}"
.format(turns, tin, ci0 + co, ci5 + co, ci9 + co, co / (ci0 + co))
)
print(
"\nReading: re-sending a growing context makes input tokens grow"
" with the"
)
print(
"square of the number of turns while output grows linearly; without"
" caching,"
)
print(
"a long agent loop is mostly paying to re-read itself. Caching"
" restores"
)
print("roughly linear cost.")
# turns input tokens cost: cost: cost: output share
# sent no cache 50 % hits 95 % hits (no cache)
# 1 2,300 0.0007 0.0007 0.0007 51.0%
# 5 18,500 0.0047 0.0036 0.0025 39.2%
# 20 179,000 0.0354 0.0220 0.0100 21.1%
# 50 972,500 0.1704 0.0958 0.0287 10.9%
# 100 3,695,000 0.6137 0.3280 0.0708 6.1%
#
# Reading: re-sending a growing context makes input tokens grow with the
# square of the number of turns while output grows linearly; without caching,
# a long agent loop is mostly paying to re-read itself. Caching restores
# roughly linear cost.输入 token 随轮数的平方增长:一轮发送 2,300 个 token,二十轮 179,000 个,一百轮 3,695,000 个,而输出只是线性增长。不做缓存时,输出占一轮对话成本的 51.0%,却只占一百轮对话的 6.1%:一个长的智能体循环,大部分钱花在重读自己上。前缀缓存使成本恢复到大致线性——缓存命中率 95% 时,一百轮对话花 0.0708,而不是 0.6137。
注
程序略去了什么。 它对长对话的每个 token 都按短上下文的价格计。按第八段程序,长上下文还会削减可行的批量并抬高输出价格,所以不精心管理上下文的长智能体循环,真实成本比这里显示的更高。
小结
对一个基于模型的产品,边际成本按使用计。预填充便宜且受算力限制;解码受内存限制,小批量时昂贵,因为每一步都重读权重。批处理分摊了这笔成本——8B 模型从批量 1 到 128 约为四十倍——直到随上下文增长的 KV 缓存占满内存、给批量封顶。多轮与智能体工作负载重发历史,所以输入随轮数的平方增长;缓存不是一种优化,而是线性成本与平方成本之间的区别。
参考文献
Fan, Y. (2026). Subscription Unit Economics, Net Revenue Retention and AI Inference Scaling Laws: Cohort Survival Mixtures, Negative-Churn Differential Equations, Token Compute Margins and Metcalfe Congestion Limits [订阅制单位经济学、净收入留存与人工智能推理标度律] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090164 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).