What does Reasoning Trajectories and Outcome Luck conclude, in numbers, about “Fingerprints under Pressure: Intraclass Correlation and Rater Agreement”?

Answer: With a single-scene coefficient of 0.30, the mean of three scenes reaches only 0.562 and the mean of six 0.720; reaching 0.70 takes 6 scenes and reaching 0.80 takes 10. With a coefficient of 0.45, three scenes give 0.711 and six give 0.831, so 3 and 5 scenes are needed respectively. With 0.60, two scenes suffice for 0.70 and three for 0.80.

Scope of the numbers: After-the-fact explanations are often folk theories: in a synthetic experiment, retrospective self-reports correlate only -0.013 with the weights used but 0.826 with an observer's guess; concurrent thinking aloud keeps order and leaps. Parameters are illustrative and all calculations come from the book's programs.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.30, 0.45, 0.562, 0.60, 0.70, 0.711

single r R(k3) R(k=6) k for 0.70 k for 0.80
0.300.562 0.720 6 10
0.450.711 0.831 3 5
0.600.818 0.900 2 3
P(order of two candidates flips), single r0.45
delta k1 k=3 k=6
0.250.436 0.391 0.348
0.500.375 0.290 0.217
1.000.261 0.134 0.059
Show the full program
# Chapter 4. p43: why one conversation is not enough.
# Spearman-Brown: reliability of the mean of k scenes is
# R_k = k r / (1 + (k - 1) r), r = single-scene ICC. Scenes needed for
# a target R: k = R (1 - r) / (r (1 - R)), rounded up.
# Order flips: two candidates whose true scores differ by delta true-
# score SDs; with reliability R the error variance is (1 - R) / R true
# variances, so P(observed order flips) = Phi(-delta / sqrt(2 (1 - R)
# / R)). Single-scene ICC values are illustrative.
import math
from statistics import NormalDist

PHI = NormalDist().cdf


def sb(r, k):
    return k * r / (1 + (k - 1) * r)


def need(r, target):
    return math.ceil(target * (1 - r) / (r * (1 - target)) - 1e-12)


def flip(delta, rel):
    return PHI(-delta / math.sqrt(2 * (1 - rel) / rel))


def main():
    print("single r  R(k=3)  R(k=6)  k for 0.70  k for 0.80")
    for r in (0.30, 0.45, 0.60):
        print("{:>8.2f}{:>8.3f}{:>8.3f}{:>11d}{:>12d}".format(
            r, sb(r, 3), sb(r, 6), need(r, 0.70), need(r, 0.80)))
    print("P(order of two candidates flips), single r = 0.45")
    print("delta   k=1    k=3    k=6")
    for delta in (0.25, 0.5, 1.0):
        print("{:<5.2f}{:>7.3f}{:>7.3f}{:>7.3f}".format(
            delta, flip(delta, sb(0.45, 1)), flip(delta, sb(0.45, 3)),
            flip(delta, sb(0.45, 6))))


main()

Single source and citation

Fan, Y. (2026). Reasoning-Trajectory Profiling, Process-Tracing Games and Separating Cognitive Structure from Stated Intentions and Outcome Luck (Version 1.0.0), Chapter 4: Fingerprints under Pressure: Intraclass Correlation and Rater Agreement. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-385 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026385,
  author    = {Fan, Yafu},
  title     = {Reasoning-Trajectory Profiling, Process-Tracing Games and Separating Cognitive Structure from Stated Intentions and Outcome Luck},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book