5D Library › Diagnostic-to-Pedagogy Conversion Funnels, Mastery-Gated Curriculum Architecture and Certification Network Economics · 中文
Chapter 8. An Audit of Course and Licence, and a Pre-Registered Experiment
Five questions
This chapter turns the book into five questions that can be put to any “reading – course – licence” design, the author’s own included. They are not business advice.
One: is the course priced below the retail value its buyers give up? If so, the course cannibalizes its buyers and must be made good by the licence funnel (Chapter 1).
Two: does the item bank hold fewer than 20 questions per attempt allowed? If so, retakes can be beaten by remembering questions (Chapter 3).
Three: is there an exam after every chapter? If so, gates and fatigue add up and completion falls sharply (Chapter 4).
Four: is the pass mark below 14/20? If so, reputation erodes, and in a two-sided network those who pass find too few clients to take the licence (Chapters 5 and 6).
Five: does the exam lack unseen transfer questions? If so, teaching to the test empties the pass mark of its filtering power (Chapters 6 and 7).
The audit as a program
The program writes the five questions as five automatic flags and applies them to two designs: a common “watch and certify” quick design (course 180, a bank of 40, 5 attempts allowed, an exam after every chapter, a pass mark of 12, no transfer questions) and the gated ladder derived in this book (course 480, a bank of 400, 4 attempts allowed, gates at subjects, a pass mark of 16, 3 transfer questions per subject).
Calculation
# Chapter 8. [p15] Five automatic flags for a course-and-licence design.
# F1 course price below the retail value its buyers give up (p02);
# F2 item bank under 20 questions per allowed attempt (p06);
# F3 an exam after every chapter rather than every subject (p07);
# F4 pass mark below 14 of 20 (p10, p11: reputation erodes);
# F5 no unseen transfer questions in the exam (p13).
# Two designs are audited: a quick "watch and certify" design and the
# gated ladder developed in this book.
DESIGNS = {
"quick": dict(price=180, buyers_pv=275.82, bank=40, attempts=5,
gate="chapter", mark=12, transfer=0),
"ladder": dict(price=480, buyers_pv=499.25, bank=400, attempts=4,
gate="subject", mark=16, transfer=3),
}
def flags(d):
return [
("F1 course below buyers' retail PV", d["price"] < d["buyers_pv"]),
("F2 bank < 20 per attempt", d["bank"] < 20 * d["attempts"]),
("F3 exam after every chapter", d["gate"] == "chapter"),
("F4 pass mark below 14/20", d["mark"] < 14),
("F5 no transfer questions", d["transfer"] == 0),
]
if __name__ == "__main__":
for name, d in DESIGNS.items():
fl = flags(d)
n = sum(f for _, f in fl)
print(" {} design: {} of 5 flags".format(name, n))
for label, f in fl:
print(" [{}] {}".format("X" if f else " ", label))
print("\nReading: the quick design trips every flag. The ladder")
print("still trips F1: priced alone the course is below what its")
print("buyers would have spent on readings, so it is justified")
print("only if the licence funnel of p01 is real - which is what")
print("the experiment of p16 must show.")
# quick design: 5 of 5 flags
# [X] F1 course below buyers' retail PV
# [X] F2 bank < 20 per attempt
# [X] F3 exam after every chapter
# [X] F4 pass mark below 14/20
# [X] F5 no transfer questions
# ladder design: 1 of 5 flags
# [X] F1 course below buyers' retail PV
# [ ] F2 bank < 20 per attempt
# [ ] F3 exam after every chapter
# [ ] F4 pass mark below 14/20
# [ ] F5 no transfer questions
#
# Reading: the quick design trips every flag. The ladder
# still trips F1: priced alone the course is below what its
# buyers would have spent on readings, so it is justified
# only if the licence funnel of p01 is real - which is what
# the experiment of p16 must show.The quick design trips all five flags. The gated ladder trips only the first: taken by itself, a course at 480 is below the retail value of 499.25 its buyers give up (Chapter 1). This is not an oversight but the book’s central judgement: the course pays only if the licence funnel is real. That is exactly what the experiment below must test.
A pre-registered experiment
Hypothesis · not established
Study protocol. Enrolees are randomised individually to an “open” arm (watching to the end counts as finishing) or a “gated” arm (subject exams, individually spaced review, transfer questions and corrective units after a failure). Primary outcome: standardised score on a set of never-seen cases on day 90, expected effect size . Harm that must be reported: completion of all 12 subjects (the model expects 0.5307 against 0.1656). Long-run registry: the share of bad sessions per session after licensing (pass mark 14 against 16; the model expects 0.00471 against 0.00333). Test: two-sided at 0.05 with power 0.90 (Cohen 1988). Carried out with pre-registered code by an independent analyst with no financial relationship with the author; results published whether positive or negative.
Calculation
# Chapter 8. [p16] Sample sizes for a pre-registered course experiment.
# Enrolees are randomised to OPEN (watch and finish) or GATE (subject
# exams, spaced review, transfer questions). Outcomes, two-sided 0.05,
# power 0.90, normal approximation:
# (a) primary: score on an unseen case set at day 90, standardised
# effect d = 0.30;
# (b) completion of all 12 subjects, 0.5307 vs 0.1656 (p08) - a harm
# the protocol must report, not hide;
# (c) bad sessions per session among licensees, 0.00471 (mark 14) vs
# 0.00333 (mark 16), p09 - counted in sessions.
import math
from statistics import NormalDist
def n_means(d, alpha=0.05, power=0.90):
z = NormalDist().inv_cdf
return math.ceil(2 * ((z(1 - alpha / 2) + z(power)) / d) ** 2)
def n_props(p1, p2, alpha=0.05, power=0.90):
z = NormalDist().inv_cdf
pb = (p1 + p2) / 2
a = z(1 - alpha / 2) * math.sqrt(2 * pb * (1 - pb))
b = z(power) * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))
return math.ceil((a + b) ** 2 / (p1 - p2) ** 2)
if __name__ == "__main__":
print(" (a) enrolees per arm, d = 0.30 : {:,d}".format(
n_means(0.30)))
print(" (b) enrolees per arm, completion : {:,d}".format(
n_props(0.5307, 0.1656)))
ns = n_props(0.00471, 0.00333)
print(" (c) sessions per arm, bad sessions : {:,d}".format(ns))
print(" = licensee-years at 100 sessions: {:,d}".format(
math.ceil(ns / 100)))
print("\nReading: the learning outcome needs a few hundred enrolees")
print("per arm; the harm to completion shows with a few dozen. The")
print("client-safety outcome needs tens of thousands of sessions")
print("per arm - hundreds of licensee-years - and can only be a")
print("long-run registry, not a trial.")
# (a) enrolees per arm, d = 0.30 : 234
# (b) enrolees per arm, completion : 34
# (c) sessions per arm, bad sessions : 44,180
# = licensee-years at 100 sessions: 442
#
# Reading: the learning outcome needs a few hundred enrolees
# per arm; the harm to completion shows with a few dozen. The
# client-safety outcome needs tens of thousands of sessions
# per arm - hundreds of licensee-years - and can only be a
# long-run registry, not a trial.The learning outcome needs 234 enrolees per arm; the difference in completion is large and shows with 34 per arm. The client-safety outcome needs 44,180 sessions per arm, about 442 licensee-years at 100 sessions a year: it cannot be one experiment, only a long-run registry. The protocol therefore makes the learning outcome primary, reports completion as a harm, and keeps client safety for a registry, rather than letting an underpowered experiment report “no difference”.
Summary
Five questions – does the course cannibalize, is the bank big enough, are gates at subjects, is the pass mark at least 14, are there transfer questions – can be written as an automatic audit. An experiment on the gated ladder should take a day-90 transfer test as its primary outcome, with 234 enrolees per arm; the fall in completion must be reported; client safety can only come from a long-run registry.
References
Fan, Y. (2026). Diagnostic-to-Pedagogy Conversion Funnels, Mastery-Gated Curriculum Architecture and Certification Network Economics: Self-Cannibalization Bounds, Spaced-Retrieval Curves and Two-Sided Practitioner Licensing [诊学一体转化漏斗、掌握度门控课程与持牌人网络经济学] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.24090197 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).