What does Reasoning Trajectories and Outcome Luck conclude, in numbers, about “Stripping Out Outcome Luck: Track Records, Sequential Tests and Multi-Method Composites”?

Answer: Records of success and failure mix the quality of decisions with luck. When sound and weak decision makers differ by a tenth in their success rates, the weak one still shows the better record about three times in ten after ten decisions, and getting that below one in ten takes more than eighty decisions; a sequential test saves about half and still needs more than a hundred. Classic validity estimates have been revised downward, and structured interviews, at the top, reach only about 0.42. Combining several methods by rule beats overall impression, and the increment from a reasoning-graph score depends on how much it overlaps with other methods; even in the most optimistic case it is modest. So outcomes are recorded separately and never enter the reasoning score, and the reasoning score is only ever one component of a composite.

Scope of the numbers: After-the-fact explanations are often folk theories: in a synthetic experiment, retrospective self-reports correlate only -0.013 with the weights used but 0.826 with an observer's guess; concurrent thinking aloud keeps order and leaps. Parameters are illustrative and all calculations come from the book's programs.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.42

anchors: structured 1998 0.51, revised 2022 0.42
validity structured conversation 0.42
validity work sample0.30
validity reasoning graph0.25
rho R opt(3) r unit(3) R opt(2) gain of graph
0.10.528 0.511 0.494 0.034
0.30.467 0.443 0.458 0.009
0.50.433 0.396 0.433 0.000
variance explained, best case rho 0.10.278
Show the full program
# Chapter 6. p63: what a multi-method composite can and cannot add.
# Validity anchors: structured interview 0.51 (Schmidt and Hunter 1998)
# and 0.42 after the range-restriction revision (Sackett et al. 2022).
# A work-sample exercise at 0.30 and a reasoning-graph score at 0.25
# are illustrative, as are the common intercorrelations 0.1 / 0.3 /
# 0.5. Optimal composite R = sqrt(v' Rxx^-1 v); unit-weighted
# composite r = sum(v) / sqrt(sum of all Rxx entries). Increment of the
# reasoning score = R(all three) - R(interview and work sample).
import math

V_SI_1998, V_SI_2022 = 0.51, 0.42
V = [V_SI_2022, 0.30, 0.25]
NAMES = ["structured conversation", "work sample", "reasoning graph"]


def solve(a, b):
    n = len(b)
    m = [row[:] + [b[i]] for i, row in enumerate(a)]
    for c in range(n):
        piv = max(range(c, n), key=lambda r: abs(m[r][c]))
        m[c], m[piv] = m[piv], m[c]
        for r in range(n):
            if r != c:
                f = m[r][c] / m[c][c]
                m[r] = [x - f * y for x, y in zip(m[r], m[c])]
    return [m[i][n] / m[i][i] for i in range(n)]


def opt_r(v, rxx):
    w = solve(rxx, v)
    return math.sqrt(math.fsum(a * b for a, b in zip(w, v)))


def unit_r(v, rxx):
    tot = math.fsum(x for row in rxx for x in row)
    return math.fsum(v) / math.sqrt(tot)


def mat(k, rho):
    return [[1.0 if i == j else rho for j in range(k)] for i in range(k)]


def main():
    print("anchors: structured 1998 {:.2f}, revised 2022 {:.2f}".format(
        V_SI_1998, V_SI_2022))
    for n, v in zip(NAMES, V):
        print("validity {:<24}{:.2f}".format(n, v))
    print("rho   R opt(3)  r unit(3)  R opt(2)  gain of graph")
    for rho in (0.1, 0.3, 0.5):
        r3, u3 = opt_r(V, mat(3, rho)), unit_r(V, mat(3, rho))
        r2 = opt_r(V[:2], mat(2, rho))
        print("{:<5.1f}{:>9.3f}{:>11.3f}{:>10.3f}{:>15.3f}".format(
            rho, r3, u3, r2, r3 - r2))
    print("variance explained, best case rho 0.1: {:.3f}".format(
        opt_r(V, mat(3, 0.1)) ** 2))


main()

Single source and citation

Fan, Y. (2026). Reasoning-Trajectory Profiling, Process-Tracing Games and Separating Cognitive Structure from Stated Intentions and Outcome Luck (Version 1.0.0), Chapter 6: Stripping Out Outcome Luck: Track Records, Sequential Tests and Multi-Method Composites. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-385 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026385,
  author    = {Fan, Yafu},
  title     = {Reasoning-Trajectory Profiling, Process-Tracing Games and Separating Cognitive Structure from Stated Intentions and Outcome Luck},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book