What does Reasoning Trajectories and Outcome Luck conclude, in numbers, about “Stripping Out Outcome Luck: Track Records, Sequential Tests and Multi-Method Composites”?
Answer: Records of success and failure mix the quality of decisions with luck. When sound and weak decision makers differ by a tenth in their success rates, the weak one still shows the better record about three times in ten after ten decisions, and getting that below one in ten takes more than eighty decisions; a sequential test saves about half and still needs more than a hundred. Classic validity estimates have been revised downward, and structured interviews, at the top, reach only about 0.42. Combining several methods by rule beats overall impression, and the increment from a reasoning-graph score depends on how much it overlaps with other methods; even in the most optimistic case it is modest. So outcomes are recorded separately and never enter the reasoning score, and the reasoning score is only ever one component of a composite.
Scope of the numbers: After-the-fact explanations are often folk theories: in a synthetic experiment, retrospective self-reports correlate only -0.013 with the weights used but 0.826 with an observer's guess; concurrent thinking aloud keeps order and leaps. Parameters are illustrative and all calculations come from the book's programs.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.42
| anchors: structured 1998 0.51, revised 2022 0.42 | |
| validity structured conversation 0.42 | |
| validity work sample | 0.30 |
| validity reasoning graph | 0.25 |
| rho R opt(3) r unit(3) R opt(2) gain of graph | |
| 0.1 | 0.528 0.511 0.494 0.034 |
| 0.3 | 0.467 0.443 0.458 0.009 |
| 0.5 | 0.433 0.396 0.433 0.000 |
| variance explained, best case rho 0.1 | 0.278 |
Show the full program
# Chapter 6. p63: what a multi-method composite can and cannot add.
# Validity anchors: structured interview 0.51 (Schmidt and Hunter 1998)
# and 0.42 after the range-restriction revision (Sackett et al. 2022).
# A work-sample exercise at 0.30 and a reasoning-graph score at 0.25
# are illustrative, as are the common intercorrelations 0.1 / 0.3 /
# 0.5. Optimal composite R = sqrt(v' Rxx^-1 v); unit-weighted
# composite r = sum(v) / sqrt(sum of all Rxx entries). Increment of the
# reasoning score = R(all three) - R(interview and work sample).
import math
V_SI_1998, V_SI_2022 = 0.51, 0.42
V = [V_SI_2022, 0.30, 0.25]
NAMES = ["structured conversation", "work sample", "reasoning graph"]
def solve(a, b):
n = len(b)
m = [row[:] + [b[i]] for i, row in enumerate(a)]
for c in range(n):
piv = max(range(c, n), key=lambda r: abs(m[r][c]))
m[c], m[piv] = m[piv], m[c]
for r in range(n):
if r != c:
f = m[r][c] / m[c][c]
m[r] = [x - f * y for x, y in zip(m[r], m[c])]
return [m[i][n] / m[i][i] for i in range(n)]
def opt_r(v, rxx):
w = solve(rxx, v)
return math.sqrt(math.fsum(a * b for a, b in zip(w, v)))
def unit_r(v, rxx):
tot = math.fsum(x for row in rxx for x in row)
return math.fsum(v) / math.sqrt(tot)
def mat(k, rho):
return [[1.0 if i == j else rho for j in range(k)] for i in range(k)]
def main():
print("anchors: structured 1998 {:.2f}, revised 2022 {:.2f}".format(
V_SI_1998, V_SI_2022))
for n, v in zip(NAMES, V):
print("validity {:<24}{:.2f}".format(n, v))
print("rho R opt(3) r unit(3) R opt(2) gain of graph")
for rho in (0.1, 0.3, 0.5):
r3, u3 = opt_r(V, mat(3, rho)), unit_r(V, mat(3, rho))
r2 = opt_r(V[:2], mat(2, rho))
print("{:<5.1f}{:>9.3f}{:>11.3f}{:>10.3f}{:>15.3f}".format(
rho, r3, u3, r2, r3 - r2))
print("variance explained, best case rho 0.1: {:.3f}".format(
opt_r(V, mat(3, 0.1)) ** 2))
main()Single source and citation
Fan, Y. (2026). Reasoning-Trajectory Profiling, Process-Tracing Games and Separating Cognitive Structure from Stated Intentions and Outcome Luck (Version 1.0.0), Chapter 6: Stripping Out Outcome Luck: Track Records, Sequential Tests and Multi-Method Composites. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-385 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026385,
author = {Fan, Yafu},
title = {Reasoning-Trajectory Profiling, Process-Tracing Games and Separating Cognitive Structure from Stated Intentions and Outcome Luck},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · The Limits of Self-Report: After-the-Fact Reasons, Thinking Aloud and Reactivity
- Chapter 2 · Four Process-Tracing Tests Written as Likelihood Ratios
- Chapter 3 · The Reasoning Graph: Nodes, Leaps, Backtracks and Revisions
- Chapter 4 · Fingerprints under Pressure: Intraclass Correlation and Rater Agreement
- Chapter 5 · The Cost of Forgery: Probing Depth, Memory Load and Bounds on Detection
- Chapter 7 · Classical Sources: the Analects on Seeing, Watching and Examining; Bayes and Aristotle on Choice
- Chapter 8 · Field Tools and Checker Specifications: the Reasoning-Trajectory Selection Ledger, the Scorer, Pre-Registration and Refutation