5D Library › Subscription Unit Economics, Net Revenue Retention and AI Inference Scaling Laws · 中文
Chapter 5. Heavy Users, Flat Prices and Routing
Routing between models
If every query is sent to the largest model, cost per query is set by the largest model. Most queries do not need it. Two designs exploit this: a router that predicts difficulty and sends only hard queries to the large model, and a cascade that tries a small model first and escalates when a checker doubts the answer (Chen et al. 2023; Ong et al. 2024). The tenth program compares them in a simple world where a large call costs 15 small ones.
Calculation
# Chapter 5. Routing between a small and a large model. Each query has a
# difficulty u, uniform on [0, 1]. The small model answers correctly when u <
# 0.70, the large one when u < 0.95. A large-model call costs 15 times a
# small-model call.
# Router: a classifier estimates u with noise (sd 0.15) and sends the query
# to the
# large model when the estimate exceeds a threshold tau.
# Cascade: always try the small model first; a checker flags a wrong small
# answer with
# probability 0.85 (and a right one with probability 0.05), and
# flagged queries are re-run on the large model (paying for both
# calls).
# 200,000 simulated queries per setting, fixed seed. Cost is per query, small
# call = 1.
import random
def router(tau, rng, n=200000):
cost = right = 0
for _ in range(n):
u = rng.random()
est = u + rng.gauss(0, 0.15)
if est > tau:
cost += 15
right += u < 0.95
else:
cost += 1
right += u < 0.70
return cost / n, right / n
def cascade(rng, catch=0.85, false_flag=0.05, n=200000):
cost = right = 0
for _ in range(n):
u = rng.random()
ok = u < 0.70
flagged = rng.random() < (false_flag if ok else catch)
cost += 1
if flagged:
cost += 15
right += u < 0.95
else:
right += ok
return cost / n, right / n
if __name__ == "__main__":
rng = random.Random(58010)
print(
" policy cost per query accuracy cost vs"
" all-large"
)
for name, c, a in (("all small", 1.0, 0.70), ("all large", 15.0, 0.95)):
print(
" {:<24} {:6.2f} {:.3f} {:5.1%}".format(
name, c, a, c / 15
)
)
for tau in (0.5, 0.6, 0.7, 0.8):
c, a = router(tau, rng)
print(" {:<24} {:6.2f} {:.3f} {:5.1%}".format(
"router, tau = {:.1f}".format(tau), c, a, c / 15))
for catch in (0.85, 0.95):
c, a = cascade(rng, catch)
print(" {:<24} {:6.2f} {:.3f} {:5.1%}".format(
"cascade, checker {:.0%}".format(catch), c, a, c / 15))
print(
"\nReading: routing buys most of the large model's accuracy for a"
" fraction of"
)
print(
"its cost, and the price of each extra point of accuracy rises"
" steeply. A"
)
print(
"cascade's value depends on how well wrong small answers are caught."
)
# policy cost per query accuracy cost vs all-large
# all small 1.00 0.700 6.7%
# all large 15.00 0.950 100.0%
# router, tau = 0.5 7.99 0.944 53.2%
# router, tau = 0.6 6.59 0.928 43.9%
# router, tau = 0.7 5.22 0.892 34.8%
# router, tau = 0.8 3.90 0.840 26.0%
# cascade, checker 85% 5.36 0.912 35.8%
# cascade, checker 95% 5.79 0.938 38.6%
#
# Reading: routing buys most of the large model's accuracy for a fraction of
# its cost, and the price of each extra point of accuracy rises steeply. A
# cascade's value depends on how well wrong small answers are caught.Routing buys most of the large model’s accuracy for a fraction of its cost. With a threshold of 0.7, the router reaches 0.892 accuracy at 34.8 % of the all-large cost; with 0.5, 0.944 at 53.2 %. Each extra point of accuracy costs more than the last: going from 0.892 to 0.944 costs more than half again as much. A cascade with a checker that catches 85 % of wrong small answers reaches 0.912 at 35.8 %; a better checker (95 %) buys 0.938 for 38.6 %. In a cascade the checker is the product.
Note
What the toy assumes. It gives the router noisy but unbiased difficulty estimates and the checker fixed catch rates. Real routers are trained on data that may not match tomorrow’s queries, and a router that silently misroutes a class of queries costs accuracy where it is least visible. The program shows the shape of the trade-off, not its size for any product.
A flat price against a heavy tail
A subscription at a flat monthly price is a bet that the average user costs much less than the price. The eleventh program prices that bet. Usage is lognormal with a median of 2 million tokens a month; the blended serving cost is 0.60 dollars per million; the price is 20 dollars.
Calculation
# Chapter 5. A flat monthly price against heavy-tailed usage. Price 20 dollars
# a month. Each user's monthly tokens are lognormal with median 2 million and
# log-sd sigma; the blended serving cost is 0.60 dollars per million tokens
# (of the order of p08's 70B figures at moderate batch). 200,000 simulated
# users per row, fixed seed. Part B adds selection: users who use less get
# less value and churn more (8 % a month below the median, 2 % above); 10,000
# new users sign up each month from the same distribution. How does gross
# margin drift over 24 months?
import math
import random
PRICE, MED, COST = 20.0, 2.0, 0.60
def users(sigma, n, rng):
return [MED * math.exp(sigma * rng.gauss(0, 1)) for _ in range(n)]
def margin(costs):
return 1 - sum(costs) / (PRICE * len(costs))
if __name__ == "__main__":
rng = random.Random(58011)
print("A. One month, flat price")
print(" sigma mean cost gross users costing share of cost:")
print(
" margin > price top 1 % top 10 %"
)
for sigma in (1.0, 1.5, 2.0, 2.5):
c = sorted(COST * u for u in users(sigma, 200000, rng))
tot = sum(c)
print(
" {:4.1f} {:8.2f} {:+7.1%} {:7.2%} {:6.1%} "
" {:6.1%}".format(
sigma,
tot / len(c),
margin(c),
sum(x > PRICE for x in c) / len(c),
sum(c[-2000:]) / tot,
sum(c[-20000:]) / tot,
)
)
print("\n sigma 2.0 with remedies:")
c = [COST * u for u in users(2.0, 200000, rng)]
cap = sorted(c)[int(0.99 * len(c))]
print(
" routing (cost x 0.35, p10) gross margin {:+.1%}".format(
margin([0.35 * x for x in c])
)
)
print(
" cap usage at the 99th percentile gross margin {:+.1%}".format(
margin([min(x, cap) for x in c])
)
)
print(
" both gross margin {:+.1%}".format(
margin([min(0.35 * x, 0.35 * cap) for x in c])
)
)
print("\nB. Selection over time (sigma 1.5): light users churn faster")
base = []
for month in range(1, 25):
base += users(1.5, 10000, rng)
if month in (1, 3, 6, 12, 24):
c = [COST * u for u in base]
print(
" month {:>2}: {:>7,} users, median usage {:.2f} M tokens,"
" gross margin {:+.1%}".format(
month, len(base), sorted(base)[len(base) // 2], margin(c)
)
)
base = [u for u in base if rng.random() > (0.08 if u < MED else 0.02)]
print("\nReading: the mean, not the median, sets the"
" cost, and the mean of a heavy")
print("tail is dominated by a few users. Flat"
" pricing then selects for them: the")
print("users who leave are the cheap ones, so"
" margin erodes with no change in")
print("price, model or behaviour.")
# A. One month, flat price
# sigma mean cost gross users costing share of cost:
# margin > price top 1 % top 10 %
# 1.0 1.97 +90.1% 0.25% 9.3% 39.0%
# 1.5 3.73 +81.4% 3.10% 20.6% 58.8%
# 2.0 8.68 +56.6% 8.03% 35.7% 75.7%
# 2.5 27.22 -36.1% 13.00% 57.1% 88.8%
#
# sigma 2.0 with remedies:
# routing (cost x 0.35, p10) gross margin +84.3%
# cap usage at the 99th percentile gross margin +65.5%
# both gross margin +87.9%
#
# B. Selection over time (sigma 1.5): light users churn faster
# month 1: 10,000 users, median usage 1.99 M tokens, gross margin +81.3%
# month 3: 28,516 users, median usage 2.13 M tokens, gross margin +80.5%
# month 6: 53,255 users, median usage 2.29 M tokens, gross margin +79.9%
# month 12: 93,557 users, median usage 2.58 M tokens, gross margin +79.0%
# month 24: 150,263 users, median usage 3.02 M tokens, gross margin +77.0%
#
# Reading: the mean, not the median, sets the cost, and the mean of a heavy
# tail is dominated by a few users. Flat pricing then selects for them: the
# users who leave are the cheap ones, so margin erodes with no change in
# price, model or behaviour.Section A shows why the median user is the wrong guide. The cost of the median user is the same in every row, but the mean cost — which sets the margin — rises with the spread of usage: 1.97 dollars at a log-sd of 1.0, 8.68 at 2.0 and 27.22 at 2.5, where the gross margin is −36.1 %. At a log-sd of 2.0, 8.03 % of users cost more than they pay, and the top 1 % of users account for 35.7 % of all serving cost. This is heavy-tail ruin: a price set for the typical user, paid for by the atypical one.
The remedies change the arithmetic, not the shape. At a log-sd of 2.0, routing (cost × 0.35, the 0.7-threshold router of the tenth program) raises the margin from +56.6 % to +84.3 %; a usage cap at the 99th percentile, to +65.5 %; both together, to +87.9 %.
Section B adds the dynamic that makes it worse over time. Users who use the product less get less from it and leave sooner. With light users churning at 8 % a month and heavy users at 2 %, the median usage of the base rises from 1.99 to 3.02 million tokens over 24 months and the gross margin falls from +81.3 % to +77.0 % — with no change in price, model or anyone’s behaviour. A flat price selects for the users it cannot afford.
Hypothesis · not established
The author’s hypothesis (not established): in AI subscription products with a flat price and no usage cap, the ratio of mean to median monthly usage among paying users rises with the age of the user base, and gross margin per user falls with it, after controlling for model and price changes. What would show it to be wrong: a stable mean-to-median ratio as the base ages.
Summary
Routing between a small and a large model buys most of the large model’s accuracy at a third to a half of its cost, with steeply rising cost for the last points of accuracy; in a cascade, the checker sets the result. Against a flat price, the mean user, not the median, sets the margin, and heavy-tailed usage can turn a healthy margin negative; a few percent of users can account for a third or more of cost. Routing and caps restore the margin, but flat pricing keeps selecting for heavy users as light ones leave, so margin erodes over time unless pricing follows usage.
References
Fan, Y. (2026). Subscription Unit Economics, Net Revenue Retention and AI Inference Scaling Laws: Cohort Survival Mixtures, Negative-Churn Differential Equations, Token Compute Margins and Metcalfe Congestion Limits [订阅制单位经济学、净收入留存与人工智能推理标度律] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090164 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).