5D Library › 订阅制单位经济学、净收入留存与人工智能推理标度律 · English

第五章 重度用户、固定价格与模型路由

在模型之间路由

如果每个查询都发给最大的模型,每个查询的成本就由最大的模型决定。大多数查询并不需要它。有两种设计利用这一点:路由器预测难度,只把难的查询发给大模型;级联先试小模型,在检查器怀疑答案时再升级 (Chen et al. 2023; Ong et al. 2024)。第十段程序在一个简单的世界里比较两者,其中一次大模型调用的成本等于 15 次小模型调用。

推导

# Chapter 5. Routing between a small and a large model. Each query has a
# difficulty u, uniform on [0, 1]. The small model answers correctly when u <
# 0.70, the large one when u < 0.95. A large-model call costs 15 times a
# small-model call.
#   Router:  a classifier estimates u with noise (sd 0.15) and sends the query
#   to the
#            large model when the estimate exceeds a threshold tau.
#   Cascade: always try the small model first; a checker flags a wrong small
#   answer with
#            probability 0.85 (and a right one with probability 0.05), and
#            flagged queries are re-run on the large model (paying for both
#            calls).
# 200,000 simulated queries per setting, fixed seed. Cost is per query, small
# call = 1.
import random

def router(tau, rng, n=200000):
    cost = right = 0
    for _ in range(n):
        u = rng.random()
        est = u + rng.gauss(0, 0.15)
        if est > tau:
            cost += 15
            right += u < 0.95
        else:
            cost += 1
            right += u < 0.70
    return cost / n, right / n

def cascade(rng, catch=0.85, false_flag=0.05, n=200000):
    cost = right = 0
    for _ in range(n):
        u = rng.random()
        ok = u < 0.70
        flagged = rng.random() < (false_flag if ok else catch)
        cost += 1
        if flagged:
            cost += 15
            right += u < 0.95
        else:
            right += ok
    return cost / n, right / n

if __name__ == "__main__":
    rng = random.Random(58010)
    print(
        "   policy                    cost per query   accuracy   cost vs"
        " all-large"
    )
    for name, c, a in (("all small", 1.0, 0.70), ("all large", 15.0, 0.95)):
        print(
            "   {:<24}    {:6.2f}         {:.3f}       {:5.1%}".format(
                name, c, a, c / 15
            )
        )
    for tau in (0.5, 0.6, 0.7, 0.8):
        c, a = router(tau, rng)
        print("   {:<24}    {:6.2f}         {:.3f}       {:5.1%}".format(
            "router, tau = {:.1f}".format(tau), c, a, c / 15))
    for catch in (0.85, 0.95):
        c, a = cascade(rng, catch)
        print("   {:<24}    {:6.2f}         {:.3f}       {:5.1%}".format(
            "cascade, checker {:.0%}".format(catch), c, a, c / 15))
    print(
        "\nReading: routing buys most of the large model's accuracy for a"
        " fraction of"
    )
    print(
        "its cost, and the price of each extra point of accuracy rises"
        " steeply. A"
    )
    print(
        "cascade's value depends on how well wrong small answers are caught."
    )
#    policy                    cost per query   accuracy   cost vs all-large
#    all small                     1.00         0.700        6.7%
#    all large                    15.00         0.950       100.0%
#    router, tau = 0.5             7.99         0.944       53.2%
#    router, tau = 0.6             6.59         0.928       43.9%
#    router, tau = 0.7             5.22         0.892       34.8%
#    router, tau = 0.8             3.90         0.840       26.0%
#    cascade, checker 85%          5.36         0.912       35.8%
#    cascade, checker 95%          5.79         0.938       38.6%
#
# Reading: routing buys most of the large model's accuracy for a fraction of
# its cost, and the price of each extra point of accuracy rises steeply. A
# cascade's value depends on how well wrong small answers are caught.

路由以一部分成本买到了大模型的大部分准确率。阈值为 0.7 时,路由器以全用大模型成本的 34.8% 达到 0.892 的准确率;阈值为 0.5 时,以 53.2% 达到 0.944。每多一个百分点的准确率都比上一个更贵:从 0.892 到 0.944,成本要再增加一半以上。一个能抓住 85% 小模型错误答案的检查器,使级联以 35.8% 的成本达到 0.912;更好的检查器(95%)以 38.6% 买到 0.938。在级联中,检查器就是产品本身。

注

这个玩具假设了什么。 它给路由器有噪声但无偏的难度估计,给检查器固定的捕获率。真实的路由器在可能与明天的查询不匹配的数据上训练,而一个悄悄把某一类查询路由错的路由器,会在最不显眼的地方损失准确率。程序展示的是权衡的形状,不是它在任何产品中的大小。

固定价格对上重尾

固定月费的订阅,是一个关于「平均用户的成本远低于价格」的赌注。第十一段程序为这个赌注定价。用量服从对数正态分布,中位数为每月 2 百万 token;综合服务成本为每百万 0.60 美元;价格为 20 美元。

推导

# Chapter 5. A flat monthly price against heavy-tailed usage. Price 20 dollars
# a month. Each user's monthly tokens are lognormal with median 2 million and
# log-sd sigma; the blended serving cost is 0.60 dollars per million tokens
# (of the order of p08's 70B figures at moderate batch). 200,000 simulated
# users per row, fixed seed. Part B adds selection: users who use less get
# less value and churn more (8 % a month below the median, 2 % above); 10,000
# new users sign up each month from the same distribution. How does gross
# margin drift over 24 months?
import math
import random

PRICE, MED, COST = 20.0, 2.0, 0.60

def users(sigma, n, rng):
    return [MED * math.exp(sigma * rng.gauss(0, 1)) for _ in range(n)]

def margin(costs):
    return 1 - sum(costs) / (PRICE * len(costs))

if __name__ == "__main__":
    rng = random.Random(58011)
    print("A. One month, flat price")
    print("   sigma   mean cost   gross    users costing   share of cost:")
    print(
        "                       margin   > price         top 1 %   top 10 %"
    )
    for sigma in (1.0, 1.5, 2.0, 2.5):
        c = sorted(COST * u for u in users(sigma, 200000, rng))
        tot = sum(c)
        print(
            "   {:4.1f}    {:8.2f}   {:+7.1%}      {:7.2%}      {:6.1%}   "
            " {:6.1%}".format(
                sigma,
                tot / len(c),
                margin(c),
                sum(x > PRICE for x in c) / len(c),
                sum(c[-2000:]) / tot,
                sum(c[-20000:]) / tot,
            )
        )
    print("\n   sigma 2.0 with remedies:")
    c = [COST * u for u in users(2.0, 200000, rng)]
    cap = sorted(c)[int(0.99 * len(c))]
    print(
        "   routing (cost x 0.35, p10)        gross margin {:+.1%}".format(
            margin([0.35 * x for x in c])
        )
    )
    print(
        "   cap usage at the 99th percentile  gross margin {:+.1%}".format(
            margin([min(x, cap) for x in c])
        )
    )
    print(
        "   both                              gross margin {:+.1%}".format(
            margin([min(0.35 * x, 0.35 * cap) for x in c])
        )
    )

    print("\nB. Selection over time (sigma 1.5): light users churn faster")
    base = []
    for month in range(1, 25):
        base += users(1.5, 10000, rng)
        if month in (1, 3, 6, 12, 24):
            c = [COST * u for u in base]
            print(
                "   month {:>2}: {:>7,} users, median usage {:.2f} M tokens,"
                " gross margin {:+.1%}".format(
                    month, len(base), sorted(base)[len(base) // 2], margin(c)
                )
            )
        base = [u for u in base if rng.random() > (0.08 if u < MED else 0.02)]
    print("\nReading: the mean, not the median, sets the"
          " cost, and the mean of a heavy")
    print("tail is dominated by a few users. Flat"
          " pricing then selects for them: the")
    print("users who leave are the cheap ones, so"
          " margin erodes with no change in")
    print("price, model or behaviour.")
# A. One month, flat price
#    sigma   mean cost   gross    users costing   share of cost:
#                        margin   > price         top 1 %   top 10 %
#     1.0        1.97    +90.1%        0.25%        9.3%     39.0%
#     1.5        3.73    +81.4%        3.10%       20.6%     58.8%
#     2.0        8.68    +56.6%        8.03%       35.7%     75.7%
#     2.5       27.22    -36.1%       13.00%       57.1%     88.8%
#
#    sigma 2.0 with remedies:
#    routing (cost x 0.35, p10)        gross margin +84.3%
#    cap usage at the 99th percentile  gross margin +65.5%
#    both                              gross margin +87.9%
#
# B. Selection over time (sigma 1.5): light users churn faster
#    month  1:  10,000 users, median usage 1.99 M tokens, gross margin +81.3%
#    month  3:  28,516 users, median usage 2.13 M tokens, gross margin +80.5%
#    month  6:  53,255 users, median usage 2.29 M tokens, gross margin +79.9%
#    month 12:  93,557 users, median usage 2.58 M tokens, gross margin +79.0%
#    month 24: 150,263 users, median usage 3.02 M tokens, gross margin +77.0%
#
# Reading: the mean, not the median, sets the cost, and the mean of a heavy
# tail is dominated by a few users. Flat pricing then selects for them: the
# users who leave are the cheap ones, so margin erodes with no change in
# price, model or behaviour.

A 段说明为什么中位用户是错误的参照。中位用户的成本在每一行都相同,但决定毛利的平均成本随用量的离散程度上升:对数标准差为 1.0 时为 1.97 美元,2.0 时为 8.68,2.5 时为 27.22,此时毛利率为 −36.1%。对数标准差为 2.0 时,8.03% 的用户成本高于他们付的钱,最重的 1% 用户占全部服务成本的 35.7%。这就是重尾破产:为典型用户定的价,由非典型用户来买单。

四种用量离散程度下,最重的用户所承担的服务成本份额:对数标准差为 2.5 时,千分之一的用户承担约四分之一的成本。

补救措施改变的是算术,不是形状。对数标准差为 2.0 时,路由(成本 × 0.35,即第十段程序中阈值 0.7 的路由器)把毛利率从 +56.6% 提高到 +84.3%;在第 99 百分位封顶用量,提高到 +65.5%;两者并用,提高到 +87.9%。

B 段加入了一个让情况随时间恶化的动态。用得少的用户从产品中得到的也少,离开得更早。当轻度用户每月流失 8%、重度用户 2% 时,24 个月里客户基数的中位用量从 1.99 百万 token 升到 3.02 百万,毛利率从 +81.3% 降到 +77.0%——价格、模型和任何人的行为都没有变化。固定价格筛选出了它负担不起的用户。

假说 · 尚未验证

作者的假说(尚未验证): 在固定价格、无用量上限的 AI 订阅产品中,控制模型与价格变化后,付费用户月用量的均值与中位数之比随用户基数的年龄上升,每用户毛利随之下降。什么结果会证明它错:基数老化时均值与中位数之比保持稳定。

小结

在小模型与大模型之间路由,以大模型三分之一到一半的成本买到其大部分准确率,最后几个百分点的准确率成本急剧上升;在级联中,检查器决定结果。面对固定价格,决定毛利的是平均用户而不是中位用户,重尾用量可以把健康的毛利率变为负数;百分之几的用户可以占三分之一以上的成本。路由与封顶能恢复毛利,但随着轻度用户离开,固定价格会持续筛选出重度用户,所以除非定价跟随用量,毛利会随时间侵蚀。

参考文献

Chen, Lingjiao, Matei Zaharia, and James Zou. 2023. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv:2305.05176.
Ong, Isaac, Amjad Almahairi, et al. 2024. RouteLLM: Learning to Route LLMs with Preference Data. arXiv:2406.18665.

Fan, Y. (2026). Subscription Unit Economics, Net Revenue Retention and AI Inference Scaling Laws: Cohort Survival Mixtures, Negative-Churn Differential Equations, Token Compute Margins and Metcalfe Congestion Limits [订阅制单位经济学、净收入留存与人工智能推理标度律] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090164 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).