5D Library › Diagnostic-to-Pedagogy Conversion Funnels, Mastery-Gated Curriculum Architecture and Certification Network Economics · 中文

Chapter 6. Two-Sided Networks, Grading Noise and Teaching to the Test

The client side: a two-sided market

Chapter 5 assumed that every licensee always holds 100 sessions a year, however many licensees there are. That holds when licensees are few and fails when they are many: clients are a finite pool, and the more licensees, the fewer clients each gets. A network of licensed practitioners is a two-sided market: clients care about practitioners’ quality and how many there are to choose from; practitioners care about how many clients they get (Rochet and Tirole 2003; Armstrong 2006).

The program lets the total sessions clients book each year rise with quality and with the number of practitioners (S=50000exp⁡(4(Q−0.75))(N/500)0.3S = 50000\exp(4(Q-0.75))(N/500)^{0.3}); practitioners keep 90% of session revenue; a person who has passed takes up the licence only if session income less the licence fee of 600 exceeds her outside option (uniform between 0 and 8,000). The number of active practitioners NN solves a fixed-point equation, found by bisection.

Calculation

# Chapter 6. [p11] Two sides: certified practitioners and their clients.
# For each pass mark T the certified pool of p09 has size N_c(T) and mean
# mastery Q(T). Clients book S = 50,000 exp(4 (Q - 0.75)) (N / 500)^0.3
# sessions a year: they want quality and some choice. Price per session
# 80 exp(4 (Q - 0.75)); the practitioner keeps 90%. A certified person
# takes up the licence (fee 600) if income S p 0.9 / N - 600 exceeds an
# outside option drawn uniform on 0..8,000, so the active number N
# solves N = N_c F(S p 0.9 / N - 600), found by bisection. Firm profit
# = 600 N + 0.1 S p. The one-sided benchmark (p09) assumes 100 sessions
# per licensee whatever their number.
import math

K = 6.0

def total_pmf(m):
    a, b = K * m, K * (1 - m)
    lb = lambda x, y: math.lgamma(x) + math.lgamma(y) - math.lgamma(x + y)
    one = [math.exp(math.lgamma(5) - math.lgamma(j + 1) - math.lgamma(5 - j)
                    + lb(j + a, 4 - j + b) - lb(a, b)) for j in range(5)]
    tot = [1.0]
    for _ in range(5):
        nxt = [0.0] * (len(tot) + 4)
        for i, x in enumerate(tot):
            for j, y in enumerate(one):
                nxt[i + j] += x * y
        tot = nxt
    return tot

def pool(T, grid=400):
    n = q = 0.0
    for i in range(grid):
        th = (i + 0.5) / grid
        w = 168 * th ** 5 * (1 - th) ** 2 / grid
        m = min(0.97, th + 0.08)
        pp = sum(total_pmf(m)[T:])
        n += w * pp
        q += w * pp * m
    return 1000 * n, q / n

def equilibrium(nc, Q):
    price = 80 * math.exp(4 * (Q - 0.75))
    def share(N):
        S = 50000 * math.exp(4 * (Q - 0.75)) * (N / 500) ** 0.3
        inc = S * price * 0.9 / N - 600
        return min(1.0, max(0.0, inc / 8000)), S
    lo, hi = 1e-6, nc
    for _ in range(80):
        mid = (lo + hi) / 2
        if nc * share(mid)[0] > mid:
            lo = mid
        else:
            hi = mid
    S = share(lo)[1]
    return lo, S, price, 600 * lo + 0.1 * S * price

if __name__ == "__main__":
    print("    T  certified  active   sessions  two-sided  one-sided")
    best2 = best1 = None
    prof2 = {}
    for T in range(10, 21):
        nc, Q = pool(T)
        N, S, p, prof = equilibrium(nc, Q)
        one = nc * (600 + 10 * p)
        prof2[T] = prof
        best2 = max(best2 or (prof, T), (prof, T))
        best1 = max(best1 or (one, T), (one, T))
        print("   {:2d} {:10.1f} {:7.1f} {:10.0f} {:10.0f} {:10.0f}".format(
            T, nc, N, S, prof, one))
    print("   best T: two-sided {}, one-sided {}".format(best2[1], best1[1]))
    print("   two-sided profit at T = 16 / at best T: {:.4f}".format(
        prof2[16] / best2[0]))
    print("\nReading: once clients are a finite pool shared among")
    print("licensees, a lax mark certifies people who then find too")
    print("few clients to take the licence; the cross-side effect moves")
    print("the best mark from 10 to 18, and the curve is flat enough")
    print("that 16/20 gives up less than one per cent.")
#     T  certified  active   sessions  two-sided  one-sided
#    10      909.4   687.1      59060     919613    1326784
#    11      868.0   697.9      61374     964040    1291977
#    12      815.2   708.0      64183    1018617    1243348
#    13      750.5   715.9      67508    1084272    1178226
#    14      674.0   674.0      69971    1120621    1094334
#    15      586.7   586.7      71314    1127620     990152
#    16      490.7   490.7      72298    1135464     865282
#    17      389.0   389.0      72607    1142861     720647
#    18      285.5   285.5      71715    1144817     558862
#    19      184.5   184.5      68583    1125667     383736
#    20       89.4    89.4      60407    1032295     198401
#    best T: two-sided 18, one-sided 10
#    two-sided profit at T = 16 / at best T: 0.9918
#
# Reading: once clients are a finite pool shared among
# licensees, a lax mark certifies people who then find too
# few clients to take the licence; the cross-side effect moves
# the best mark from 10 to 18, and the curve is flat enough
# that 16/20 gives up less than one per cent.
One-sided static profit and two-sided equilibrium profit against the pass mark (each scaled by its own maximum). The dotted line is 16.

The result turns Chapter 5’s static conclusion around. With a mark of 10, 909.4 pass but only 687.1 take up the licence: the client pool is spread thin, and many who pass get too few clients to make the fee worthwhile. As the mark rises, fewer pass but each gets more clients and the quality premium rises; from 14 upward every passer takes the licence. The most profitable mark in the two-sided equilibrium is 18, and profit at 16 is 0.9918 of the optimum, less than one per cent short. The profit curve is very flat between 14 and 19.

This section is the result in the book most favourable to the author’s outline: once the client side is treated as a finite pool that is spread thin, 16/20 is close to optimal. But it rests on the elasticity of client demand with respect to the number of practitioners (0.3 here) and on the distribution of outside options, both of which need real data.

Noise in AI grading

The false passes and false fails of Chapter 3 come from the coarseness of five questions. AI grading adds a second layer of noise: the program lets the grader mark a question one point too high or too low with probability ee (half each way, kept between 0 and 4), and compares a “borderline review” policy, under which scripts graded 15 or 16 go to a person who marks them without error.

Calculation

# Chapter 6. [p12] A noisy AI grader and a human look at the borderline.
# Exam and population as in p05 (5 questions x 4 points, pass 16, point
# correlation k = 6, candidates Beta(6, 3), competent m >= 0.80). The
# grader mis-scores each question with probability e by one point, up
# or down with equal chance (clipped to 0..4). The joint law of the true
# and the graded total is convolved exactly over the 5 questions.
# Policy B (borderline review): a script graded 15 or 16 goes to a
# human who scores it without error. Reported: pass rate, false passes
# among passers, and the competent who fail.
import math

K = 6.0

def bb(m):
    a, b = K * m, K * (1 - m)
    lb = lambda x, y: math.lgamma(x) + math.lgamma(y) - math.lgamma(x + y)
    return [math.exp(math.lgamma(5) - math.lgamma(j + 1) - math.lgamma(5 - j)
                     + lb(j + a, 4 - j + b) - lb(a, b)) for j in range(5)]

def joint(m, e):
    one = {}
    for j, p in enumerate(bb(m)):
        for d, w in ((0, 1 - e), (-1, e / 2), (1, e / 2)):
            g = min(4, max(0, j + d))
            one[(j, g)] = one.get((j, g), 0.0) + p * w
    tot = {(0, 0): 1.0}
    for _ in range(5):
        nxt = {}
        for (t, g), p in tot.items():
            for (j, h), q in one.items():
                key = (t + j, g + h)
                nxt[key] = nxt.get(key, 0.0) + p * q
        tot = nxt
    return tot

def pass_prob(m, e, review):
    s = 0.0
    for (t, g), p in joint(m, e).items():
        if review and g in (15, 16):
            s += p * (t >= 16)
        else:
            s += p * (g >= 16)
    return s

def metrics(e, review, n=400):
    pas = fp = fn = comp = 0.0
    for i in range(n):
        m = (i + 0.5) / n
        w = 168 * m ** 5 * (1 - m) ** 2 / n
        pp = pass_prob(m, e, review)
        pas += w * pp
        if m >= 0.80:
            comp += w
            fn += w * (1 - pp)
        else:
            fp += w * pp
    return pas, fp / pas, fn / comp

if __name__ == "__main__":
    print("     e  review   pass rate  false pass  competent fail")
    for e in (0.0, 0.1, 0.2, 0.3):
        for rv in (False, True):
            if e == 0.0 and rv:
                continue
            a, b, c = metrics(e, rv)
            print("   {:.1f} {:7s} {:11.4f} {:11.4f} {:15.4f}".format(
                e, "yes" if rv else "no", a, b, c))
    print("\nReading: a question marked 4 can only be marked down, so")
    print("grader noise mostly fails competent candidates and hardly")
    print("changes the share of weak passers. A human review of scripts")
    print("graded 15 or 16 recovers most of the damage. The exam's own")
    print("coarseness (e = 0) remains the larger error.")
#      e  review   pass rate  false pass  competent fail
#    0.0 no           0.3223      0.4930          0.1953
#    0.1 no           0.3145      0.4935          0.2156
#    0.1 yes          0.3211      0.4925          0.1977
#    0.2 no           0.3066      0.4944          0.2367
#    0.2 yes          0.3179      0.4914          0.2039
#    0.3 no           0.2986      0.4957          0.2585
#    0.3 yes          0.3134      0.4900          0.2131
#
# Reading: a question marked 4 can only be marked down, so
# grader noise mostly fails competent candidates and hardly
# changes the share of weak passers. A human review of scripts
# graded 15 or 16 recovers most of the damage. The exam's own
# coarseness (e = 0) remains the larger error.

A surprising result: symmetric grading noise does not mainly let weak candidates through; it fails strong ones. The reason is the edge of the scale: a question with full marks of 4 can only be marked down, never up. At e=0.3e = 0.3, the share of the competent who fail on the day rises from 0.1953 to 0.2585, while the false-pass share barely moves (0.4957). Borderline review brings false fails of the competent back to 0.2131.

So AI grading noise is not the main problem here; the coarseness of five questions is (at e=0e = 0 false passes are already 0.4930). Borderline review is a cheap patch with a clear effect.

Teaching to the test

Campbell observed that the more a quantitative indicator is used for social decisions, the more it is subject to corruption and the more it distorts the processes it was meant to monitor (Campbell 1979). Exams are no exception: once licence income depends on an exam, coaching aimed at the item bank appears, and the correlation between score and real skill falls.

The program describes this with a Gaussian model: exam performance correlates with real skill at ρ\rho, the pass mark always lets 32.23% through (the pass rate of Chapter 3), and the brand price follows the formula of Chapter 5.

Calculation

# Chapter 6. [p13] Teaching to the test: when the score stops tracking skill.
# Skill z and exam aptitude x are standard normal with correlation rho:
# x = rho z + sqrt(1 - rho^2) u. Mastery is m = 0.667 + 0.15 z (about
# the Beta(6, 3) of p05). The pass mark is set so that 32.23% pass, the
# p05 pass rate. Coaching aimed at the item bank lowers rho. For each
# rho the program reports (200,000 draws, fixed seed) the mean mastery
# of passers, the share of passers below 0.80, and the brand price of
# p09, 80 exp(4 (Q - 0.75)). A remedy: add k unseen transfer questions
# whose own correlation with skill is 0.8, averaged with the old score.
import math
import random

def sim(rho, rng, n=200000, transfer=0, rt=0.8):
    xs, ms = [], []
    for _ in range(n):
        z = rng.gauss(0, 1)
        x = rho * z + math.sqrt(1 - rho * rho) * rng.gauss(0, 1)
        if transfer:
            t = sum(rt * z + math.sqrt(1 - rt * rt) * rng.gauss(0, 1)
                    for _ in range(transfer)) / transfer
            x = (x + transfer * t) / (1 + transfer)
        xs.append(x)
        ms.append(0.667 + 0.15 * z)
    cut = sorted(xs)[int(n * (1 - 0.3223))]
    pm = [m for x, m in zip(xs, ms) if x >= cut]
    q = sum(pm) / len(pm)
    low = sum(m < 0.80 for m in pm) / len(pm)
    return q, low, 80 * math.exp(4 * (q - 0.75))

if __name__ == "__main__":
    rng = random.Random(74013)
    print("   rho  transfer   mean m  below 0.80  brand price")
    for rho in (0.9, 0.7, 0.5, 0.3):
        q, low, p = sim(rho, rng)
        print("   {:.1f} {:9d} {:8.4f} {:11.4f} {:12.2f}".format(
            rho, 0, q, low, p))
    for k in (1, 3):
        q, low, p = sim(0.3, rng, transfer=k)
        print("   {:.1f} {:9d} {:8.4f} {:11.4f} {:12.2f}".format(
            0.3, k, q, low, p))
    print("\nReading: as coaching decouples the score from skill, the")
    print("pass pool drifts toward the population mean and the brand")
    print("premium that justified the mark melts away. Unseen transfer")
    print("questions - one corner shown, three to be found - restore")
    print("most of it.")
#    rho  transfer   mean m  below 0.80  brand price
#    0.9         0   0.8177      0.4561       104.89
#    0.7         0   0.7839      0.5603        91.60
#    0.5         0   0.7509      0.6422        80.27
#    0.3         0   0.7176      0.7158        70.28
#    0.3         1   0.7841      0.5591        91.68
#    0.3         3   0.8150      0.4650       103.75
#
# Reading: as coaching decouples the score from skill, the
# pass pool drifts toward the population mean and the brand
# premium that justified the mark melts away. Unseen transfer
# questions - one corner shown, three to be found - restore
# most of it.

As ρ\rho falls from 0.9 to 0.3, passers’ mean mastery falls from 0.8177 to 0.7176, the share of passers below 0.80 rises from 0.4561 to 0.7158, and the brand price falls from 104.89 to 70.28. The mark is still there and the pass rate has not changed, but what it filters out is no longer low skill.

The remedy is to add transfer questions the candidate has never seen: give one example and ask her to work out the other cases. Even when the old questions have been coached down to ρ=0.3\rho = 0.3, one transfer question with correlation 0.8 brings mean mastery back to 0.7841, and three bring it back to 0.8150, close to the level at ρ=0.9\rho = 0.9. This is exactly the meaning of “shown one corner, come back with the other three” in Chapter 7.

Hypothesis · not established

The author’s interpretation (not yet tested): in the author’s examinations, at least 3 of each subject’s 5 questions should be new cases from outside the bank (giving the method, not a similar worked example), which would keep the quality of the licensed group stable even after a coaching market appears. What would show this wrong: if real exam data showed that scores on bank questions and on new cases correlate equally with client ratings after licensing, transfer questions would measure skill no better and the requirement could be dropped.

Summary

Once the client side is treated as a finite pool, the best mark in the two-sided equilibrium is 18, and 16 is less than one per cent short. AI grading noise mainly hurts strong candidates, and borderline review repairs it. Teaching to the test empties the pass mark of its filtering power; unseen transfer questions bring it back.

References

Armstrong, Mark. 2006. “Competition in Two-Sided Markets.” RAND Journal of Economics 37 (3): 668–91.
Campbell, Donald T. 1979. “Assessing the Impact of Planned Social Change.” Evaluation and Program Planning 2 (1): 67–90.
Rochet, Jean-Charles, and Jean Tirole. 2003. “Platform Competition in Two-Sided Markets.” Journal of the European Economic Association 1 (4): 990–1029.

Fan, Y. (2026). Diagnostic-to-Pedagogy Conversion Funnels, Mastery-Gated Curriculum Architecture and Certification Network Economics: Self-Cannibalization Bounds, Spaced-Retrieval Curves and Two-Sided Practitioner Licensing [诊学一体转化漏斗、掌握度门控课程与持牌人网络经济学] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090197 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).