5D Library › Mechanism Design, Vickrey–Clarke–Groves Auctions and Incentive-Compatible Community Engineering · 中文
Chapter 8 Practice and Pre-Registered Testing
Note
The checklist and experimental protocol in this chapter are for research and self-examination; they are not legal, business, marketing or platform-compliance advice. The checklist cannot tell you which rule to adopt; it can only help you write down your assumptions before you decide. All community, bidding and identity data in the book are fictional or simulated.
Part 1 A four-step checklist
Chapters 1 to 7 can be condensed into a four-step checklist for examining an auction, bidding or community incentive rule. The thresholds in it are examples of rules the user should write down in advance, not recommended values.
- Information and commitment. What private information does the rule ask participants to reveal? Is truth-telling a dominant strategy, only optimal in the Bayesian sense, or not an equilibrium at all (Chapters 1 and 3)? Can the rule-maker commit not to revise the rule afterwards or apply it retroactively (the Coase problem of Chapter 1)? If not, participants will wait or conceal.
- Externalities and payments. Does each person pay the loss he imposes on others (Chapter 2)? Is the outcome in the core, or do the seller or losers have reasons to strike private deals around the rule? With a GSP-type rule, is it understood that truth-telling is not an equilibrium and revenue depends on equilibrium selection (Chapter 3)? If revenue is the goal, is the value distribution regular, and is a reserve or ironing needed (Chapter 4)?
- Identities and false names. Can one person take part under several identities? For every class of declarable types, does the rule meet the conditions for false-name-proofness (Chapter 2)? Is the reputation function symmetric (Chapter 6)? Under the author’s cost-boundary model, does each identity cost more than the average gain from one more identity?
- Budget and inflation. Must the rule balance its budget? If so, what do the impossibility theorems of Chapter 5 say must be given up? Does the issuance of points grow faster than the real supply of rewards, and can the sinks (redemption, expiry) only lower the level rather than change the growth rate (Chapter 6)?
A fictional example. A fictional learning community with 3,000 members auctions 20 one-to-one mentoring slots for points each month, gives extra rewards to the top ten “contributors”, and scores contribution by “total likes received”. Applying the checklist: in step 1, the auction is first-price sealed-bid, so truth-telling is not an equilibrium, and the moderators often cut prices after unsold auctions, so members have learnt to wait (the Coase problem); in step 2, switching to a second-price auction or an ascending clock auction would make truthful bidding a dominant strategy, and the latter is easier to understand; in step 3, “total likes” is a symmetric reputation function, and by the program of Chapter 6 a few second accounts liking one another reach the top ten, so it should be replaced by recommendation flow from trusted members, or identity costs should be raised; in step 4, points are issued per post while mentoring slots do not increase, so points will lose value at the rate of membership growth, and issuance should be tied to the supply of rewards. Every change has a cost, and the reasons for each trade-off should be written down in advance.
Part 2 A pre-registered, incentivised laboratory experiment
Question and design choices
In laboratory position auctions in which both rules are explained truthfully and bids have real monetary consequences, does VCG raise the share of truthful bids compared with GSP, by at least the smallest effect of interest?
The author’s framework suggests comparing GSP and VCG with either “the share of rounds in which the bid equals the value (within a tolerance)” or “efficiency” as the primary outcome. The book chooses the former, for three reasons. First, it tests directly the core claim of Chapters 1 and 3: VCG makes truth-telling a dominant strategy and GSP does not; the experiment asks how large this theoretical difference is in behaviour, and the evidence cited in Chapter 1 shows that people do not always see dominant strategies (Kagel et al. 1987; Li 2017). Second, efficiency does not separate the two rules well: in theory both are efficient in locally envy-free equilibria, and laboratory evidence shows similar revenue and efficiency (Fukuda et al. 2013). Third, in the design below the rivals are computer programs bidding by a disclosed rule, so allocative efficiency depends mainly on whether the participant yields the slot to programs with higher values; it is better suited as a secondary outcome.
One point needs stating: in GSP truthful bidding is usually not the participant’s best response, so “a lower truthful share in the GSP arm” does not by itself show irrationality; the protocol therefore makes “the share of bids within the best-response interval” a secondary outcome, to separate “failing to see the dominant strategy” from “correctly departing from the truth”.
Protocol
| Element | Specification |
|---|---|
| Who runs it; recruitment | University laboratories unconnected with the author and his businesses, at least two; adults recruited from each laboratory’s regular subject pool; no recruitment from the author’s community, students or customers, and no recruitment notice through any of the author’s channels |
| Software | An open-source experimental platform (for example oTree (Chen et al. 2016)); code and configuration published at registration |
| Task | In each round two slots (100 and 60 clicks per period); the participant bids against two computer rivals; the three values per click are drawn independently each round, uniform on [1, 20] and rounded to 0.01; bids are entered in units of 0.01 (at a value of 1 the ±5 % tolerance is ±0.05, still 11 admissible bids); the participant knows only his own value; the computer rivals bid by a publicly explained rule (bid equal to their own value); 3 practice rounds, then 30 scored rounds |
| Arms | GSP: rank by bid, pay the next bid; VCG: rank by bid, pay the clicks one takes from others. Both arms’ instructions describe the payment rule truthfully and completely, with worked examples in the same format and a comprehension quiz; no deception |
| Randomisation | Individual 1:1 allocation, blocked within session and stratified by laboratory; the allocation sequence generated and sealed by a third party |
| Blinding | Participants are not told the hypothesis nor that another rule exists; the data analyst sees only “A / B” labels until the analysis code is locked |
| Payment | A show-up fee plus the earnings of 3 randomly drawn scored rounds, converted to cash at a rate announced in advance |
| Primary outcome | For each participant, the share of the 30 scored rounds in which the bid is within 5 % of the value; effect = mean in the VCG arm − mean in the GSP arm, intention to treat, 95 % confidence interval from a Welch-type standard error |
| Pre-specified qualifier | If the truthful share in the VCG arm is below 0.5, the report must state that “most VCG bids were still not truthful” |
| Secondary outcomes | Share of bids within the best-response interval; efficiency of slot allocation; participants’ earnings and auctioneer revenue; mean relative deviation ; learning over rounds; comprehension scores; all descriptive or exploratory |
Guarding against data dredging
There is one primary outcome, one tolerance fixed in advance (5 %) and one method of analysis; the protocol, software and analysis code are registered publicly before the first participant is enrolled, with the code’s hash registered alongside (Nosek et al. 2018); secondary and exploratory results are clearly labelled and cannot replace the primary conclusion. Book 33 discussed the statistical rationale for these practices.
Sample size
The sample size requires assumptions about the truthful shares in the two arms and about individual differences. This protocol’s planning assumptions (not data) are: a mean share of 0.25 in the GSP arm; individual true shares with standard deviation 0.25 (beta-distributed), with binomial sampling error over 30 rounds on top; and a smallest effect of interest of 10 percentage points. The reason for 10 points is that a smaller difference would not justify changing practice to a payment rule that is harder to explain; the “obviously strategy-proof” research mentioned in Chapter 1 compared clock and sealed-bid auctions, and its figures cannot be borrowed here.
Calculation
# Chapter 8. Sample size for the pre-registered laboratory experiment. Participants are
# randomised individually to GSP or VCG position auctions (2 slots, 2 computer rivals
# whose bidding rule is disclosed), 30 scored rounds with fresh induced values.
# Primary outcome per participant: share of scored rounds with |bid - value| <= 5 % of
# value. Effect = mean share (VCG) - mean share (GSP). Planning assumptions (NOT data):
# GSP mean 0.25; SESOI 0.10; individual true shares vary with sd 0.25 (beta
# distributed); binomial noise over 30 rounds on top. Two-sided alpha 0.05, power 90 %.
import math, random, statistics
Z = statistics.NormalDist().inv_cdf
ROUNDS = 30
def total_sd(m, sd):
within = (m * (1 - m) - sd ** 2) / ROUNDS # E[p(1-p)] / rounds
return math.sqrt(sd ** 2 + within)
def n_per_arm(m0, delta, sd):
s0, s1 = total_sd(m0, sd), total_sd(m0 + delta, sd)
return math.ceil((Z(0.975) + Z(0.90)) ** 2 * (s0 ** 2 + s1 ** 2) / delta ** 2)
def beta_params(m, sd):
k = m * (1 - m) / sd ** 2 - 1
return m * k, (1 - m) * k
def simulate(n, m0, delta, sd, reps, rng):
hits = 0
for _ in range(reps):
arms = []
for m in (m0, m0 + delta):
a, b = beta_params(m, sd); ys = []
for _ in range(n):
p = rng.betavariate(a, b)
ys.append(sum(rng.random() < p for _ in range(ROUNDS)) / ROUNDS)
arms.append(ys)
d = statistics.fmean(arms[1]) - statistics.fmean(arms[0])
se = math.sqrt((statistics.variance(arms[0]) + statistics.variance(arms[1])) / n)
hits += d - Z(0.975) * se > 0
return hits / reps
if __name__ == "__main__":
m0, delta, sd = 0.25, 0.10, 0.25
n = n_per_arm(m0, delta, sd)
print(f"total sd of a participant's share: GSP {total_sd(m0, sd):.4f},"
f" VCG {total_sd(m0 + delta, sd):.4f}")
print(f"normal approximation: {n} analysable participants per arm ({2 * n} in all)")
rng = random.Random(50)
pw = simulate(n, m0, delta, sd, 1500, rng)
print(f"simulated power (beta-binomial, Welch-type z test, 1,500 trials): {pw:.3f}"
f" (MC s.e. {math.sqrt(pw * (1 - pw) / 1500):.3f})")
print(f"simulated false-positive rate at no effect: "
f"{simulate(n, m0 + 0.05, 0.0, sd, 1500, rng):.3f} (one-sided, nominal 0.025)")
plan = math.ceil(n / 10) * 10
pw2 = simulate(plan, m0, delta, sd, 1500, rng)
print(f"planned: round up to {plan} analysable per arm; simulated power {pw2:.3f}")
rec = math.ceil(plan / 0.9 / 2) * 2
print(f"allowing 10 % non-completion: randomise {rec} per arm, {2 * rec} in all")
print("sensitivity, analysable per arm SESOI: 0.08 0.10 0.12")
for s in (0.20, 0.25, 0.30):
print(f" sd of individual shares {s:.2f} " +
"".join(f"{n_per_arm(m0, d, s):8d}" for d in (0.08, 0.10, 0.12)))
print(f" GSP mean 0.40 instead of 0.25, sd 0.25 " +
"".join(f"{n_per_arm(0.40, d, 0.25):8d}" for d in (0.08, 0.10, 0.12)))
# total sd of a participant's share: GSP 0.2582, VCG 0.2608
# normal approximation: 142 analysable participants per arm (284 in all)
# simulated power (beta-binomial, Welch-type z test, 1,500 trials): 0.891 (MC s.e. 0.008)
# simulated false-positive rate at no effect: 0.024 (one-sided, nominal 0.025)
# planned: round up to 150 analysable per arm; simulated power 0.913
# allowing 10 % non-completion: randomise 168 per arm, 336 in all
# sensitivity, analysable per arm SESOI: 0.08 0.10 0.12
# sd of individual shares 0.20 150 96 67
# sd of individual shares 0.25 221 142 99
# sd of individual shares 0.30 309 198 138
# GSP mean 0.40 instead of 0.25, sd 0.25 226 145 101The total standard deviation of a participant’s share is 0.2582 and 0.2608 in the two arms. The normal approximation needs 142 analysable participants per arm; a beta-binomial simulation (1,500 trials) gives a power of 0.891 (Monte Carlo standard error 0.008), consistent with 0.90; with no effect the one-sided false-positive rate is 0.024, close to the nominal 0.025. The plan rounds up to 150 analysable per arm (simulated power 0.913) and, allowing for 10 % non-completion, randomises 168 per arm, 336 in all. The sensitivity table shows that if the standard deviation of individual shares is 0.30, an effect of 10 points needs 198 per arm; a smallest effect of interest of 8 points needs 221; changing the GSP baseline from 0.25 to 0.40 changes little (145). The right panel of Figure 5 shows power curves under three assumptions about individual differences.
Following the discussion in Chapter 6 of Book 33, “90 % power” means that if the true effect equals the smallest effect of interest exactly, the probability of reaching some conclusion that “VCG raises the share of truthful bids” (a confidence interval whose lower bound is above 0) is about 90 %; in that case the strongest conclusion, “raised by at least the smallest effect of interest”, has only about a 2.5 % chance of appearing.
Decision rule
Calculation
# Decision rule, fixed before unblinding. D = mean truthful-bid share (VCG arm) - mean
# truthful-bid share (GSP arm), per-participant shares over the 30 scored rounds,
# intention to treat; 95 % CI from the Welch-type standard error. SESOI 0.10.
# Gates, checked first: comprehension software or payment errors in more than 5 % of
# sessions; non-completion above 20 % in either arm or differing by more than 10
# points; arm labels revealed to the analyst before the analysis code was locked
# -> NOT INTERPRETABLE. Pre-specified qualifier: if the VCG arm's truthful share is
# below 0.5, the report adds that most VCG bids were still not truthful.
SESOI = 0.10
def verdict(d, lo, hi, vcg_share, errors=0.01, drop=(0.08, 0.09), unblinded=False):
for bad, why in ((errors > 0.05, f"errors in {errors:.0%} of sessions (limit 5 %)"),
(max(drop) > 0.20 or abs(drop[0] - drop[1]) > 0.10,
f"non-completion {drop[0]:.0%} / {drop[1]:.0%}"),
(unblinded, "labels revealed before the code was locked")):
if bad:
return f"D = {d:+.3f}: NOT INTERPRETABLE - {why}"
if lo >= SESOI: v = "VCG raises truthful bidding by at least the SESOI"
elif lo > 0: v = "VCG raises truthful bidding; not shown to reach the SESOI"
elif hi < 0: v = "GSP has the higher truthful share"
elif lo >= -SESOI and hi <= SESOI: v = "no difference worth acting on"
else: v = "unclear: the interval is too wide to decide"
if vcg_share < 0.5:
v += f"\n qualifier: only {vcg_share:.0%} of VCG-arm bids were truthful"
return f"D = {d:+.3f} ({lo:+.3f} to {hi:+.3f}): {v}"
for case in ((0.25, 0.19, 0.31, 0.62), (0.12, 0.05, 0.19, 0.38),
(0.02, -0.04, 0.08, 0.30), (-0.08, -0.14, -0.02, 0.22),
(0.06, -0.03, 0.15, 0.33), (0.25, 0.19, 0.31, 0.62, 0.08),
(0.25, 0.19, 0.31, 0.62, 0.01, (0.25, 0.10))):
print(verdict(*case))
print("secondary outcomes (efficiency of slot allocation, payoffs, revenue, mean")
print(" |bid - value| / value, learning over rounds, comprehension scores) are")
print(" reported with intervals; they never decide the verdict")
# D = +0.250 (+0.190 to +0.310): VCG raises truthful bidding by at least the SESOI
# D = +0.120 (+0.050 to +0.190): VCG raises truthful bidding; not shown to reach the SESOI
# qualifier: only 38% of VCG-arm bids were truthful
# D = +0.020 (-0.040 to +0.080): no difference worth acting on
# qualifier: only 30% of VCG-arm bids were truthful
# D = -0.080 (-0.140 to -0.020): GSP has the higher truthful share
# qualifier: only 22% of VCG-arm bids were truthful
# D = +0.060 (-0.030 to +0.150): unclear: the interval is too wide to decide
# qualifier: only 33% of VCG-arm bids were truthful
# D = +0.250: NOT INTERPRETABLE - errors in 8% of sessions (limit 5 %)
# D = +0.250: NOT INTERPRETABLE - non-completion 25% / 10%
# secondary outcomes (efficiency of slot allocation, payoffs, revenue, mean
# |bid - value| / value, learning over rounds, comprehension scores) are
# reported with intervals; they never decide the verdictThe example results are made up. If any precondition fails (software or payment errors in more than 5 % of sessions; non-completion above 20 % in either arm, or differing between arms by more than 10 points; labels revealed before the analysis code is locked), the report says “not interpretable” whatever the primary outcome. Only when the lower bound of the confidence interval is at least 0.10 does the report say “VCG raises truthful bidding by at least the smallest effect of interest”; when the lower bound is above 0 but below 0.10, “VCG raises truthful bidding, but the size is not established”; when the upper bound is below 0, “the GSP arm has the higher truthful share”; when the interval lies within ±0.10, “no difference worth changing practice for”; otherwise, “cannot decide”. The qualifier does not change the category of the verdict but is appended to it: in the second example, for instance, only 38 % of VCG-arm bids are truthful, and the report must say so.
What this experiment cannot show
First, students with small stakes in a laboratory are not real advertisers: real advertisers have budgets, long-term relationships and professional tools, and the conclusions cannot be carried over directly to real platforms. Second, the rivals are computer programs bidding by a disclosed rule; against human rivals, equilibrium selection and learning in GSP could be entirely different. Third, it tests one configuration (two slots, two rivals, one value distribution, 30 rounds) and does not test Chapter 3’s results on revenue. Fourth, “a bid within 5 % of the value” is an operational definition; with a different tolerance the shares would differ, which is why the tolerance is fixed in advance and cannot be chosen afterwards. Fifth, it does not test the points, badge or Sybil models of Chapter 6; those need cluster-randomised trials in independent communities. Sixth, even if the VCG arm bids truthfully more often, that does not show that platforms should switch to VCG: the problems of revenue, the core, false names and budgets discussed in Chapters 2 and 5 remain. Seventh, if the result is “no difference worth changing practice for”, that is the experiment’s conclusion, whatever the author may have claimed.
Ethics and conflicts of interest
The protocol must be approved by the ethics committee of the institution of each laboratory that runs it; participants give informed consent and may withdraw at any time while keeping the show-up fee; there is no deception, and both rules are explained truthfully; payments are no lower than local standards for similar studies; data are stored anonymously, and summary data and code are published as registered. The results will be published whether positive or negative. To limit the influence of the author’s interests: the experiment is run and analysed by independent researchers, does not recruit from the author’s community, students or customers, the author has no access to the randomisation sequence or the raw data, and the protocol and code are registered publicly before enrolment.
Disclosure
The author uses the works of 5D Library, including this book, as the methodological basis of his community courses, the 5D Ascension Cards (Dimensional Shift Card Decks) and the Yafu AI knowledge base, and therefore has a commercial interest in the subject of this book. The book promotes no course, card deck or service; the Chapter 8 trial is designed to be run and analysed by researchers independent of the author’s business and does not recruit from the author’s community, students or customers; all the code and the study protocol are published openly for anyone to check. The author runs online learning communities and designs their points and incentive rules, and founded and operates Asia Influencer X and Yafu Global (whose influencer-marketing business involves price quotes, bidding and reputation ratings between creators and brands), and therefore has a direct commercial interest in the subject of this book; the book uses no data from the author’s communities or businesses as evidence and does not promote their services.
Chapter summary
The four-step checklist examines an incentive rule from four sides: information and commitment, externalities and payments, identities and false names, budget and inflation; the fictional example shows how one community’s auction, ranking and points rules can fall into all four problems at once. The pre-registered laboratory experiment randomises participants to GSP or VCG position auctions, with a single primary outcome (the share of rounds with truthful bids), a smallest effect of interest of 10 percentage points, 150 analysable and 168 randomised participants per arm for 90 % power, independent researchers, a decision rule written as code and a pre-specified qualifier for “most VCG bids still untruthful”; it will be published whatever the result. It tests only the behavioural consequence of one difference between rules in the laboratory, cannot prove the book’s other models, and cannot be carried over directly to real platforms.
References
Fan, Y. (2026). Mechanism Design, Vickrey–Clarke–Groves Auctions and Incentive-Compatible Community Engineering: From Myerson's Optimal Auction and the Hurwicz Impossibility Theorem to Sybil-Resistant Reputation Games, Read Beside Classical Legalist Treatises on Rewards and Incentives [机制设计、维克里—克拉克—格罗夫斯(VCG)拍卖与数字社群激励相容工程] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23836274 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).