5D Library › 订阅制单位经济学、净收入留存与人工智能推理标度律 · English

第四章 一个 Token 的成本:预填充、解码与内存墙

为什么 AI 软件的成本结构不同

传统软件的边际成本接近于零:多服务一个用户,只多花一点带宽和存储。建立在大语言模型之上的软件,每一次使用都有边际成本:每一个回答都在加速器上计算,成本随用户读写的多少而变化。对传统订阅软件来说又高又稳定的毛利率,变成了用量、模型规模、批处理与上下文长度的函数。本章从第一性原理计算这个函数;第五章说明它对固定月费的订阅价格做了什么。

证据

一次前向计算的算术。 一个有 NN 个参数的稠密 Transformer,每个 token 的前向计算约需 2N2N 次浮点运算 (Kaplan et al. 2020);决定模型规模的标度律文献,也意味着服务成本随规模增长 (Hoffmann et al. 2022)。推理服务有两个瓶颈不同的阶段。读取提示词(预填充)并行处理大量 token,受算力限制。写出回答(解码)每一步为批中的每个序列产生一个 token,在小批量时每一步受内存带宽限制:需要读取全部权重,外加每个序列注意力的键-值(KV)缓存 (Pope et al. 2023)。分组查询注意力通过共享键与值的头来缩小 KV 缓存 (Ainslie et al. 2023),分页式内存管理让更多序列共享加速器的内存 (Kwon et al. 2023)。

一百万个 token 的成本

第八段程序使用一个示意性的加速器——每秒 101510^{15} 次浮点运算、每秒 3×10123\times10^{12} 字节的带宽、80 GB 内存、每小时 2.00 美元——以及两种示意性的稠密模型规模。这些数字不是任何产品的报价;它们说明成本如何随规模变化。

推导

# Chapter 4. What a token costs to serve. An illustrative accelerator: 1.0e15
# dense FLOP/s, 3.0e12 bytes/s of memory bandwidth, 80e9 bytes of memory, 2.00
# dollars an hour. Two illustrative dense transformer sizes with grouped-query
# attention, weights in 16-bit (2 bytes per parameter):
#   "8B":  8e9 parameters, 32 layers, 8 KV heads of dimension 128, on 1 device
#   "70B": 70e9 parameters, 80 layers, 8 KV heads of dimension 128, on 4
#   devices
# KV cache per token = 2 (K and V) x layers x KV heads x head dim x 2 bytes.
# PREFILL (reading the prompt) is compute-bound: about 2 x params FLOPs per
# token, at 50 % utilisation. DECODE (writing the answer) runs one step per
# output token for the whole batch; each step must read all weights plus every
# sequence's KV cache from memory, and also do 2 x params FLOPs per sequence;
# the step takes whichever is longer.
def model(name):
    if name == "8B":
        return dict(P=8e9, L=32, H=8, Dh=128, dev=1)
    return dict(P=70e9, L=80, H=8, Dh=128, dev=4)

FLOPS, BW, MEM, PRICE = 1.0e15, 3.0e12, 80e9, 2.00

def kv_bytes(m):
    return 2 * m["L"] * m["H"] * m["Dh"] * 2

def prefill_cost_per_m(m):
    sec = 2 * m["P"] * 1e6 / (FLOPS * m["dev"] * 0.5)
    return sec * PRICE * m["dev"] / 3600

def decode(m, batch, ctx):
    w = 2 * m["P"]
    kv = batch * ctx * kv_bytes(m)
    if w + kv > MEM * m["dev"]:
        return None
    t_mem = (w + kv) / (BW * m["dev"])
    t_flop = 2 * m["P"] * batch / (FLOPS * m["dev"])
    t = max(t_mem, t_flop)
    return (
        t / batch * 1e6 * PRICE * m["dev"] / 3600,
        (w + kv) / (MEM * m["dev"]),
        kv / (w + kv),
    )

if __name__ == "__main__":
    for name in ("8B", "70B"):
        m = model(name)
        print("{}: KV cache {:,} bytes per token; prefill {:.3f} dollars per"
              .format(name, kv_bytes(m), prefill_cost_per_m(m)))
        print("   million input tokens")
        print(
            "   batch   context   decode $/M      memory used   KV share of"
        )
        print("                     output tokens                 bytes read")
        for ctx in (2000, 32000):
            for batch in (1, 8, 32, 128):
                r = decode(m, batch, ctx)
                if r is None:
                    print(
                        "   {:>5}   {:>6}    does not fit in memory".format(
                            batch, ctx
                        )
                    )
                else:
                    print(
                        "   {:>5}   {:>6}    {:9.3f}          {:6.1%}       "
                        " {:6.1%}".format(batch, ctx, r[0], r[1], r[2])
                    )
        print()
    print(
        "Reading: output tokens cost far more than input tokens at small"
        " batches,"
    )
    print(
        "because every decode step re-reads the weights. Batching spreads"
        " that read"
    )
    print(
        "over many users until the KV cache - which grows with context -"
        " takes over"
    )
    print(
        "the memory and the bandwidth. Long contexts cap the batch, and so"
        " the price."
    )
# 8B: KV cache 131,072 bytes per token; prefill 0.018 dollars per
#    million input tokens
#    batch   context   decode $/M      memory used   KV share of
#                      output tokens                 bytes read
#        1     2000        3.012           20.3%          1.6%
#        8     2000        0.419           22.6%         11.6%
#       32     2000        0.141           30.5%         34.4%
#      128     2000        0.072           61.9%         67.7%
#        1    32000        3.740           25.2%         20.8%
#        8    32000        1.147           61.9%         67.7%
#       32    32000    does not fit in memory
#      128    32000    does not fit in memory
#
# 70B: KV cache 327,680 bytes per token; prefill 0.156 dollars per
#    million input tokens
#    batch   context   decode $/M      memory used   KV share of
#                      output tokens                 bytes read
#        1     2000       26.047           44.0%          0.5%
#        8     2000        3.362           45.4%          3.6%
#       32     2000        0.932           50.3%         13.0%
#      128     2000        0.324           70.0%         37.5%
#        1    32000       27.868           47.0%          7.0%
#        8    32000        5.183           70.0%         37.5%
#       32    32000    does not fit in memory
#      128    32000    does not fit in memory
#
# Reading: output tokens cost far more than input tokens at small batches,
# because every decode step re-reads the weights. Batching spreads that read
# over many users until the KV cache - which grows with context - takes over
# the memory and the bandwidth. Long contexts cap the batch, and so the price.

由此得出三个事实。

小批量时输出比输入贵得多。 对 8B 模型,预填充一百万个输入 token 花 0.018 美元,而批量为 1 时一百万个输出 token 花 3.012 美元:每个解码步都要重新读取全部权重,才为每个序列产生一个 token。

毛利在批处理里。 上下文为 2,000 个 token 时,把批量从 1 提到 128,8B 模型的输出成本从每百万 token 3.012 美元降到 0.072,70B 模型从 26.047 降到 0.324。权重每步只读一次,由批中的每个序列分摊。

上下文长度给批量封顶。 KV 缓存随上下文的每个 token 增长:8B 模型每个 token 131,072 字节,70B 模型 327,680 字节。上下文为 32,000 个 token 时,8B 模型装不下批量 32,批量为 8 时输出成本为 1.147,而短上下文时为 0.419;70B 模型只装得下 8 个序列。长上下文不只是要读的输入更多;它拿走了让输出变便宜的批处理。

两种示意模型在 2,000 与 32,000 个 token 上下文下,每百万输出 token 的解码成本随批量的变化。曲线在批量装不进内存处中止。

智能体在反复重读自己

第九段程序把这些单价用于多轮工作。一段对话或一个智能体循环,每一轮都把全部历史作为输入重新发送。

推导

# Chapter 4. Why multi-turn and agent workloads cost more than their output
# suggests. Unit prices are taken from p08's 70B rows at batch 32 and 2,000
# tokens of context: 0.156 dollars per million input (prefill) tokens and
# 0.932 per million output tokens. A conversation starts with a 2,000-token
# system prompt and tools description; each turn adds 300 tokens from the user
# (or tool results) and 400 generated tokens, and the whole conversation so
# far is sent again as input on the next turn. With prefix caching only the
# tokens not already processed are prefilled; a cache hit rate h means a share
# h of the previously seen prefix is found in cache (the rest is prefilled
# again).
IN, OUT = 0.156e-6, 0.932e-6
SYS, USER, GEN = 2000, 300, 400

def cost(turns, h):
    ctx, c_in, c_out, toks_in = SYS, 0.0, 0.0, 0
    for k in range(turns):
        new = (SYS if k == 0 else 0) + USER
        seen = ctx - (SYS if k == 0 else 0)
        prefill = new + (1 - h) * (seen if k > 0 else 0)
        toks_in += ctx + USER if k > 0 else SYS + USER
        c_in += prefill * IN
        c_out += GEN * OUT
        ctx += USER + GEN
    return c_in, c_out, toks_in

if __name__ == "__main__":
    print(
        "   turns   input tokens   cost:       cost:        cost:       "
        " output share"
    )
    print(
        "           sent           no cache    50 % hits    95 % hits    (no"
        " cache)"
    )
    for turns in (1, 5, 20, 50, 100):
        ci0, co, tin = cost(turns, 0.0)
        ci5, _, _ = cost(turns, 0.5)
        ci9, _, _ = cost(turns, 0.95)
        print(
            "   {:>5}   {:>12,}   {:9.4f}   {:9.4f}    {:9.4f}       {:6.1%}"
            .format(turns, tin, ci0 + co, ci5 + co, ci9 + co, co / (ci0 + co))
        )
    print(
        "\nReading: re-sending a growing context makes input tokens grow"
        " with the"
    )
    print(
        "square of the number of turns while output grows linearly; without"
        " caching,"
    )
    print(
        "a long agent loop is mostly paying to re-read itself. Caching"
        " restores"
    )
    print("roughly linear cost.")
#    turns   input tokens   cost:       cost:        cost:        output share
#            sent           no cache    50 % hits    95 % hits    (no cache)
#        1          2,300      0.0007      0.0007       0.0007        51.0%
#        5         18,500      0.0047      0.0036       0.0025        39.2%
#       20        179,000      0.0354      0.0220       0.0100        21.1%
#       50        972,500      0.1704      0.0958       0.0287        10.9%
#      100      3,695,000      0.6137      0.3280       0.0708         6.1%
#
# Reading: re-sending a growing context makes input tokens grow with the
# square of the number of turns while output grows linearly; without caching,
# a long agent loop is mostly paying to re-read itself. Caching restores
# roughly linear cost.

输入 token 随轮数的平方增长:一轮发送 2,300 个 token,二十轮 179,000 个,一百轮 3,695,000 个,而输出只是线性增长。不做缓存时,输出占一轮对话成本的 51.0%,却只占一百轮对话的 6.1%:一个长的智能体循环,大部分钱花在重读自己上。前缀缓存使成本恢复到大致线性——缓存命中率 95% 时,一百轮对话花 0.0708,而不是 0.6137。

注

程序略去了什么。 它对长对话的每个 token 都按短上下文的价格计。按第八段程序,长上下文还会削减可行的批量并抬高输出价格,所以不精心管理上下文的长智能体循环,真实成本比这里显示的更高。

小结

对一个基于模型的产品,边际成本按使用计。预填充便宜且受算力限制;解码受内存限制,小批量时昂贵,因为每一步都重读权重。批处理分摊了这笔成本——8B 模型从批量 1 到 128 约为四十倍——直到随上下文增长的 KV 缓存占满内存、给批量封顶。多轮与智能体工作负载重发历史,所以输入随轮数的平方增长;缓存不是一种优化,而是线性成本与平方成本之间的区别。

参考文献

Ainslie, Joshua, James Lee-Thorp, et al. 2023. “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.” Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.
Hoffmann, Jordan, Sebastian Borgeaud, et al. 2022. Training Compute-Optimal Large Language Models. arXiv:2203.15556.
Kaplan, Jared, Sam McCandlish, et al. 2020. Scaling Laws for Neural Language Models. arXiv:2001.08361.
Kwon, Woosuk, Zhuohan Li, et al. 2023. “Efficient Memory Management for Large Language Model Serving with PagedAttention.” Proceedings of the 29th Symposium on Operating Systems Principles.
Pope, Reiner, Sholto Douglas, et al. 2023. “Efficiently Scaling Transformer Inference.” Proceedings of Machine Learning and Systems 5.

Fan, Y. (2026). Subscription Unit Economics, Net Revenue Retention and AI Inference Scaling Laws: Cohort Survival Mixtures, Negative-Churn Differential Equations, Token Compute Margins and Metcalfe Congestion Limits [订阅制单位经济学、净收入留存与人工智能推理标度律] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090164 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).