5D Library › Diagnostic-to-Pedagogy Conversion Funnels, Mastery-Gated Curriculum Architecture and Certification Network Economics · 中文

Chapter 8. An Audit of Course and Licence, and a Pre-Registered Experiment

Five questions

This chapter turns the book into five questions that can be put to any “reading – course – licence” design, the author’s own included. They are not business advice.

One: is the course priced below the retail value its buyers give up? If so, the course cannibalizes its buyers and must be made good by the licence funnel (Chapter 1).

Two: does the item bank hold fewer than 20 questions per attempt allowed? If so, retakes can be beaten by remembering questions (Chapter 3).

Three: is there an exam after every chapter? If so, gates and fatigue add up and completion falls sharply (Chapter 4).

Four: is the pass mark below 14/20? If so, reputation erodes, and in a two-sided network those who pass find too few clients to take the licence (Chapters 5 and 6).

Five: does the exam lack unseen transfer questions? If so, teaching to the test empties the pass mark of its filtering power (Chapters 6 and 7).

The audit as a program

The program writes the five questions as five automatic flags and applies them to two designs: a common “watch and certify” quick design (course 180, a bank of 40, 5 attempts allowed, an exam after every chapter, a pass mark of 12, no transfer questions) and the gated ladder derived in this book (course 480, a bank of 400, 4 attempts allowed, gates at subjects, a pass mark of 16, 3 transfer questions per subject).

Calculation

# Chapter 8. [p15] Five automatic flags for a course-and-licence design.
# F1 course price below the retail value its buyers give up (p02);
# F2 item bank under 20 questions per allowed attempt (p06);
# F3 an exam after every chapter rather than every subject (p07);
# F4 pass mark below 14 of 20 (p10, p11: reputation erodes);
# F5 no unseen transfer questions in the exam (p13).
# Two designs are audited: a quick "watch and certify" design and the
# gated ladder developed in this book.

DESIGNS = {
    "quick": dict(price=180, buyers_pv=275.82, bank=40, attempts=5,
                  gate="chapter", mark=12, transfer=0),
    "ladder": dict(price=480, buyers_pv=499.25, bank=400, attempts=4,
                   gate="subject", mark=16, transfer=3),
}

def flags(d):
    return [
        ("F1 course below buyers' retail PV", d["price"] < d["buyers_pv"]),
        ("F2 bank < 20 per attempt", d["bank"] < 20 * d["attempts"]),
        ("F3 exam after every chapter", d["gate"] == "chapter"),
        ("F4 pass mark below 14/20", d["mark"] < 14),
        ("F5 no transfer questions", d["transfer"] == 0),
    ]

if __name__ == "__main__":
    for name, d in DESIGNS.items():
        fl = flags(d)
        n = sum(f for _, f in fl)
        print("   {} design: {} of 5 flags".format(name, n))
        for label, f in fl:
            print("     [{}] {}".format("X" if f else " ", label))
    print("\nReading: the quick design trips every flag. The ladder")
    print("still trips F1: priced alone the course is below what its")
    print("buyers would have spent on readings, so it is justified")
    print("only if the licence funnel of p01 is real - which is what")
    print("the experiment of p16 must show.")
#    quick design: 5 of 5 flags
#      [X] F1 course below buyers' retail PV
#      [X] F2 bank < 20 per attempt
#      [X] F3 exam after every chapter
#      [X] F4 pass mark below 14/20
#      [X] F5 no transfer questions
#    ladder design: 1 of 5 flags
#      [X] F1 course below buyers' retail PV
#      [ ] F2 bank < 20 per attempt
#      [ ] F3 exam after every chapter
#      [ ] F4 pass mark below 14/20
#      [ ] F5 no transfer questions
#
# Reading: the quick design trips every flag. The ladder
# still trips F1: priced alone the course is below what its
# buyers would have spent on readings, so it is justified
# only if the licence funnel of p01 is real - which is what
# the experiment of p16 must show.

The quick design trips all five flags. The gated ladder trips only the first: taken by itself, a course at 480 is below the retail value of 499.25 its buyers give up (Chapter 1). This is not an oversight but the book’s central judgement: the course pays only if the licence funnel is real. That is exactly what the experiment below must test.

A pre-registered experiment

Hypothesis · not established

Study protocol. Enrolees are randomised individually to an “open” arm (watching to the end counts as finishing) or a “gated” arm (subject exams, individually spaced review, transfer questions and corrective units after a failure). Primary outcome: standardised score on a set of never-seen cases on day 90, expected effect size d=0.30d = 0.30. Harm that must be reported: completion of all 12 subjects (the model expects 0.5307 against 0.1656). Long-run registry: the share of bad sessions per session after licensing (pass mark 14 against 16; the model expects 0.00471 against 0.00333). Test: two-sided at 0.05 with power 0.90 (Cohen 1988). Carried out with pre-registered code by an independent analyst with no financial relationship with the author; results published whether positive or negative.

Calculation

# Chapter 8. [p16] Sample sizes for a pre-registered course experiment.
# Enrolees are randomised to OPEN (watch and finish) or GATE (subject
# exams, spaced review, transfer questions). Outcomes, two-sided 0.05,
# power 0.90, normal approximation:
# (a) primary: score on an unseen case set at day 90, standardised
#     effect d = 0.30;
# (b) completion of all 12 subjects, 0.5307 vs 0.1656 (p08) - a harm
#     the protocol must report, not hide;
# (c) bad sessions per session among licensees, 0.00471 (mark 14) vs
#     0.00333 (mark 16), p09 - counted in sessions.
import math
from statistics import NormalDist

def n_means(d, alpha=0.05, power=0.90):
    z = NormalDist().inv_cdf
    return math.ceil(2 * ((z(1 - alpha / 2) + z(power)) / d) ** 2)

def n_props(p1, p2, alpha=0.05, power=0.90):
    z = NormalDist().inv_cdf
    pb = (p1 + p2) / 2
    a = z(1 - alpha / 2) * math.sqrt(2 * pb * (1 - pb))
    b = z(power) * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))
    return math.ceil((a + b) ** 2 / (p1 - p2) ** 2)

if __name__ == "__main__":
    print("   (a) enrolees per arm, d = 0.30      : {:,d}".format(
        n_means(0.30)))
    print("   (b) enrolees per arm, completion    : {:,d}".format(
        n_props(0.5307, 0.1656)))
    ns = n_props(0.00471, 0.00333)
    print("   (c) sessions per arm, bad sessions  : {:,d}".format(ns))
    print("       = licensee-years at 100 sessions: {:,d}".format(
        math.ceil(ns / 100)))
    print("\nReading: the learning outcome needs a few hundred enrolees")
    print("per arm; the harm to completion shows with a few dozen. The")
    print("client-safety outcome needs tens of thousands of sessions")
    print("per arm - hundreds of licensee-years - and can only be a")
    print("long-run registry, not a trial.")
#    (a) enrolees per arm, d = 0.30      : 234
#    (b) enrolees per arm, completion    : 34
#    (c) sessions per arm, bad sessions  : 44,180
#        = licensee-years at 100 sessions: 442
#
# Reading: the learning outcome needs a few hundred enrolees
# per arm; the harm to completion shows with a few dozen. The
# client-safety outcome needs tens of thousands of sessions
# per arm - hundreds of licensee-years - and can only be a
# long-run registry, not a trial.

The learning outcome needs 234 enrolees per arm; the difference in completion is large and shows with 34 per arm. The client-safety outcome needs 44,180 sessions per arm, about 442 licensee-years at 100 sessions a year: it cannot be one experiment, only a long-run registry. The protocol therefore makes the learning outcome primary, reports completion as a harm, and keeps client safety for a registry, rather than letting an underpowered experiment report “no difference”.

Summary

Five questions – does the course cannibalize, is the bank big enough, are gates at subjects, is the pass mark at least 14, are there transfer questions – can be written as an automatic audit. An experiment on the gated ladder should take a day-90 transfer test as its primary outcome, with 234 enrolees per arm; the fall in completion must be reported; client safety can only come from a long-run registry.

References

Cohen, Jacob. 1988. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Lawrence Erlbaum.

Fan, Y. (2026). Diagnostic-to-Pedagogy Conversion Funnels, Mastery-Gated Curriculum Architecture and Certification Network Economics: Self-Cannibalization Bounds, Spaced-Retrieval Curves and Two-Sided Practitioner Licensing [诊学一体转化漏斗、掌握度门控课程与持牌人网络经济学] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090197 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).