What does Reasoning Trajectories and Outcome Luck conclude, in numbers, about “Fingerprints under Pressure: Intraclass Correlation and Rater Agreement”?
Answer: With a single-scene coefficient of 0.30, the mean of three scenes reaches only 0.562 and the mean of six 0.720; reaching 0.70 takes 6 scenes and reaching 0.80 takes 10. With a coefficient of 0.45, three scenes give 0.711 and six give 0.831, so 3 and 5 scenes are needed respectively. With 0.60, two scenes suffice for 0.70 and three for 0.80.
Scope of the numbers: After-the-fact explanations are often folk theories: in a synthetic experiment, retrospective self-reports correlate only -0.013 with the weights used but 0.826 with an observer's guess; concurrent thinking aloud keeps order and leaps. Parameters are illustrative and all calculations come from the book's programs.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.30, 0.45, 0.562, 0.60, 0.70, 0.711
| single r R(k | 3) R(k=6) k for 0.70 k for 0.80 |
| 0.30 | 0.562 0.720 6 10 |
| 0.45 | 0.711 0.831 3 5 |
| 0.60 | 0.818 0.900 2 3 |
| P(order of two candidates flips), single r | 0.45 |
| delta k | 1 k=3 k=6 |
| 0.25 | 0.436 0.391 0.348 |
| 0.50 | 0.375 0.290 0.217 |
| 1.00 | 0.261 0.134 0.059 |
Show the full program
# Chapter 4. p43: why one conversation is not enough.
# Spearman-Brown: reliability of the mean of k scenes is
# R_k = k r / (1 + (k - 1) r), r = single-scene ICC. Scenes needed for
# a target R: k = R (1 - r) / (r (1 - R)), rounded up.
# Order flips: two candidates whose true scores differ by delta true-
# score SDs; with reliability R the error variance is (1 - R) / R true
# variances, so P(observed order flips) = Phi(-delta / sqrt(2 (1 - R)
# / R)). Single-scene ICC values are illustrative.
import math
from statistics import NormalDist
PHI = NormalDist().cdf
def sb(r, k):
return k * r / (1 + (k - 1) * r)
def need(r, target):
return math.ceil(target * (1 - r) / (r * (1 - target)) - 1e-12)
def flip(delta, rel):
return PHI(-delta / math.sqrt(2 * (1 - rel) / rel))
def main():
print("single r R(k=3) R(k=6) k for 0.70 k for 0.80")
for r in (0.30, 0.45, 0.60):
print("{:>8.2f}{:>8.3f}{:>8.3f}{:>11d}{:>12d}".format(
r, sb(r, 3), sb(r, 6), need(r, 0.70), need(r, 0.80)))
print("P(order of two candidates flips), single r = 0.45")
print("delta k=1 k=3 k=6")
for delta in (0.25, 0.5, 1.0):
print("{:<5.2f}{:>7.3f}{:>7.3f}{:>7.3f}".format(
delta, flip(delta, sb(0.45, 1)), flip(delta, sb(0.45, 3)),
flip(delta, sb(0.45, 6))))
main()Single source and citation
Fan, Y. (2026). Reasoning-Trajectory Profiling, Process-Tracing Games and Separating Cognitive Structure from Stated Intentions and Outcome Luck (Version 1.0.0), Chapter 4: Fingerprints under Pressure: Intraclass Correlation and Rater Agreement. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-385 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026385,
author = {Fan, Yafu},
title = {Reasoning-Trajectory Profiling, Process-Tracing Games and Separating Cognitive Structure from Stated Intentions and Outcome Luck},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · The Limits of Self-Report: After-the-Fact Reasons, Thinking Aloud and Reactivity
- Chapter 2 · Four Process-Tracing Tests Written as Likelihood Ratios
- Chapter 3 · The Reasoning Graph: Nodes, Leaps, Backtracks and Revisions
- Chapter 5 · The Cost of Forgery: Probing Depth, Memory Load and Bounds on Detection
- Chapter 6 · Stripping Out Outcome Luck: Track Records, Sequential Tests and Multi-Method Composites
- Chapter 7 · Classical Sources: the Analects on Seeing, Watching and Examining; Bayes and Aristotle on Choice
- Chapter 8 · Field Tools and Checker Specifications: the Reasoning-Trajectory Selection Ledger, the Scorer, Pre-Registration and Refutation