Chapter 8 Practice: A Prompt Audit and a Pre-registered Trial

Part 1 A four-step audit

Before asking a model for help with a piece of research, an article or a plan, it is worth auditing the brief. The four steps below turn the ideas of the earlier chapters into counts that can be made in a few minutes.

Constraints. Count the constraints that remove degrees of freedom from the answer: scope, audience, length, form, what is out of bounds, what must be included. (Chapter 1: information redirects probability.)
Bridges. Count the explicit bridges between named fields, each stating the relation to be examined, for example “use the delayed-feedback model from control theory to analyse pitch correction in singing”. (Chapter 2: a query’s reach.)
Falsifiers. Count the conditions stated in advance that would show an idea in the answer to be wrong. (Chapter 4: protection against seeing what one expects.)
Independent sources. Count the independent sources, data or tests against which the answer will be checked. (Chapter 3: the variety of the checking system.)

Calculation

The audit as a score out of 20, with each step capped so that none dominates, for three briefs.

# The four-step audit of a prompt or research brief, scored by hand and summed.
# Each count is capped so that no single step can dominate.
def audit(constraints, bridges, falsifiers, sources):
    parts = {
        "1 constraints that remove degrees of freedom": min(constraints, 5),
        "2 explicit bridges between named fields":      min(bridges, 3) * 5 / 3,
        "3 conditions that would show the idea wrong":  min(falsifiers, 3) * 5 / 3,
        "4 independent sources to check against":       min(sources, 3) * 5 / 3,
    }
    return parts, sum(parts.values())

briefs = {
    "'Write about AI and creativity.'": (0, 0, 0, 0),
    "domain brief (scope, audience, length, 2 sources)": (4, 0, 0, 2),
    "cross-field brief (as above + 2 bridges + 2 tests)": (4, 2, 2, 3),
}
for name, counts in briefs.items():
    parts, total = audit(*counts)
    print(f"{name:52s} score {total:4.1f} / 20  ("
          + ", ".join(f"{v:.1f}" for v in parts.values()) + ")")
# 'Write about AI and creativity.'                     score  0.0 / 20  (0.0, 0.0, 0.0, 0.0)
# domain brief (scope, audience, length, 2 sources)    score  7.3 / 20  (4.0, 0.0, 0.0, 3.3)
# cross-field brief (as above + 2 bridges + 2 tests)   score 15.7 / 20  (4.0, 3.3, 3.3, 5.0)

The score is a teaching device, not a validated scale; its weights are a choice. Whether a higher score predicts better outputs is exactly what the trial below tests.

Note

More is not always better. A brief overloaded with constraints can leave no room for a good answer, and bridges between fields that have no real relation produce confident nonsense. The audit is a check for missing structure, not a target to maximise.

Part 2 A pre-registered randomised trial

Question

For research-planning tasks, do cross-field structured prompts lead a language model to produce research proposals that blinded experts rate as more novel and more rigorous than proposals produced from conventional single-domain prompts?

Design

Element	Specification
Units	60 research questions sampled in advance from a public list of open problems across six broad areas, ten per area
Prompts	For each question, two prompts written by independent prompt writers who do not see the outputs: A, a conventional single-domain prompt (the question, its field, the requested output); B, a cross-field structured prompt (the same, plus at least two named field bridges, two falsifiable conditions and three independent sources to check against). A is padded with neutral background so that both prompts have similar length
Generation	One fixed model version and settings, recorded; three samples per prompt, one chosen at random for rating
Order	Pairs presented in random order; outputs stripped of formatting cues that would reveal the arm
Raters	Three experts per area, blind to the arm and to the hypothesis, rate each proposal on novelty and rigour (1–7, with written anchors)
Outcomes	Co-primary: mean rated novelty and mean rated rigour; secondary: rater agreement (ICC), proportion of proposals with at least one factual error found by a fact-checker
Analysis	Mixed model with question as a random effect; each co-primary outcome tested at α = 0.025

Sample size

Calculation

import math
from statistics import NormalDist

z = NormalDist().inv_cdf

def topics_needed(sd_diff, delta, alpha=0.025, power=0.90):
    """Paired design: each research topic gets both prompt types. alpha = 0.025 per
    co-primary outcome (novelty, rigour), a Bonferroni split of 0.05."""
    za, zb = z(1 - alpha / 2), z(power)
    return math.ceil(((za + zb) * sd_diff / delta) ** 2 + za ** 2 / 2)

# Outcomes: mean of three blinded experts' ratings on a 1–7 scale.
SESOI = 0.5
for sd in (0.8, 1.0, 1.2):
    print(f"SD of within-topic difference {sd}: {topics_needed(sd, SESOI)} topics")
# SD of within-topic difference 0.8: 35 topics
# SD of within-topic difference 1.0: 53 topics
# SD of within-topic difference 1.2: 75 topics

The protocol plans for the middle case, 53 questions, and samples 60 to allow for questions dropped because a model refuses or a rater cannot assess them. The standard deviation of 1.0 rating point for within-question differences is an assumption to be checked in a 6-question pilot.

Decision rule

Calculation

SESOI = 0.5     # rating points on a 1–7 scale

def verdict(name, diff, lo, hi):
    """diff = cross-field structured prompt minus single-domain prompt, with a
    97.5 % confidence interval. Fixed before any data are seen."""
    if lo >= SESOI:
        v = "clearly higher (by at least 0.5)"
    elif lo > 0:
        v = "higher; size not shown to reach 0.5"
    elif hi < 0:
        v = "lower"
    elif lo >= -SESOI and hi <= SESOI:
        v = "no difference worth acting on"
    else:
        v = "unclear"
    return f"{name:8s} {diff:+.2f} ({lo:+.2f} to {hi:+.2f}): {v}"

print(verdict("novelty", 0.9, 0.6, 1.2))
print(verdict("rigour", 0.1, -0.3, 0.5))
print("Both outcomes are reported together: higher novelty with no gain in rigour is a")
print("different result from higher novelty and higher rigour, and is reported as such.")
# novelty  +0.90 (+0.60 to +1.20): clearly higher (by at least 0.5)
# rigour   +0.10 (-0.30 to +0.50): no difference worth acting on
# Both outcomes are reported together: higher novelty with no gain in rigour is a
# different result from higher novelty and higher rigour, and is reported as such.

The examples are invented, to show in advance how results will be reported. Novelty and rigour are reported together, because a gain in novelty without a gain in rigour is a different and less useful result.

What the trial cannot show

The trial compares two kinds of prompt for one model, one version and one kind of task. A result may not transfer to other models, which change quickly, or to other tasks. Prompt B differs from A in several ways at once; a positive result supports the package, not any one of its parts. And expert ratings of novelty are not the same as importance: a proposal can be new and still not worth doing.

Ethics and conflicts of interest

The trial involves expert raters but no patients or vulnerable participants; raters give informed consent and are paid for their time. The protocol will be publicly registered before any outputs are generated, following the principles of pre-registration (Nosek et al. 2018).

Disclosure

The author founded and runs Asia Influencer X and Yafu Global, uses AI tools in influencer marketing, digital marketing and content production, and advocates the human-architect, AI-drafting and human-verification workflow described in this book, and so has an interest in a favourable result. To limit the influence of that interest: the protocol, the prompt templates, the analysis code and the decision rule are published before any outputs are generated; the prompt writers, raters and analyst are independent of the author’s businesses; and the results, outputs, ratings and code will be published whatever they show.

References

Nosek, Brian A., Charles R. Ebersole, Alexander C. DeHaven, and David T. Mellor. 2018. “The Preregistration Revolution.” Proceedings of the National Academy of Sciences 115 (11): 2600–2606.

Fan, Y. (2026). The Algorithmic Mirror: Self-Attention Geometry, Ashby's Requisite Variety and Cognitive Projection in Information Theory and East Asian Philosophy of Mind [算法之镜] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23723557 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).