What does Reasoning Trajectories and Outcome Luck conclude, in numbers, about “The Limits of Self-Report: After-the-Fact Reasons, Thinking Aloud and Reactivity”?
Answer: Thinking aloud keeps 0.919 of the steps, its Kendall correlation with the real order is 0.977, and 0.968 of reports still show at least one leap. Retrospective recall keeps only 0.648 of the steps, its correlation falls to 0.833, and only 0.235 of reports still show a leap.
Scope of the numbers: After-the-fact explanations are often folk theories: in a synthetic experiment, retrospective self-reports correlate only -0.013 with the weights used but 0.826 with an observer's guess; concurrent thinking aloud keeps order and leaps. Parameters are illustrative and all calculations come from the book's programs.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.235, 0.648, 0.833, 0.919, 0.968, 0.977
| synthetic people per condition: 2,000 | |||
| condition | coverage | tau(real) | leap kept |
| concurrent | 0.919 | 0.977 | 0.968 |
| retrospective | 0.648 | 0.833 | 0.235 |
| textbook order tau(real) = 0.786 |
Show the full program
# Chapter 1. p12: does the report keep the order and the leaps?
# A reasoning episode has eight steps 0..7. The textbook order is 0..7;
# the real episode jumped twice: 4 came before 2 and 3, and 6 before 5.
# Concurrent think-aloud: each step reported with prob 0.92, neighbours
# swapped with prob 0.04. Retrospective account: each step recalled with
# prob 0.65; with prob 0.6 the person rebuilds the textbook order,
# otherwise keeps the real order with neighbour swaps at 0.10.
# Measures over 2,000 synthetic people per condition: coverage, Kendall
# tau against the real order, and the share of reports that still show
# a leap (4 before 2, or 6 before 5). Parameters are illustrative.
import random
import statistics
REAL = [0, 1, 4, 2, 3, 6, 5, 7]
N = 2000
def swaps(seq, p, rng):
s = list(seq)
for i in range(len(s) - 1):
if rng.random() < p:
s[i], s[i + 1] = s[i + 1], s[i]
return s
def tau(rep):
pos = {v: i for i, v in enumerate(REAL)}
conc = disc = 0
for i in range(len(rep)):
for j in range(i + 1, len(rep)):
if pos[rep[i]] < pos[rep[j]]:
conc += 1
else:
disc += 1
tot = conc + disc
return (conc - disc) / tot if tot else 0.0
def keeps_leap(rep):
where = {v: i for i, v in enumerate(rep)}
for a, b in ((4, 2), (6, 5)):
if a in where and b in where and where[a] < where[b]:
return 1
return 0
def concurrent(rng):
kept = [s for s in REAL if rng.random() < 0.92]
return swaps(kept, 0.04, rng)
def retrospective(rng):
kept = [s for s in REAL if rng.random() < 0.65]
if rng.random() < 0.6:
return sorted(kept)
return swaps(kept, 0.10, rng)
def main():
rng = random.Random(38512)
print("synthetic people per condition: {:,}".format(N))
print("condition coverage tau(real) leap kept")
for name, fn in (("concurrent", concurrent),
("retrospective", retrospective)):
cov, taus, leap = [], [], 0
for _ in range(N):
rep = fn(rng)
cov.append(len(rep) / len(REAL))
taus.append(tau(rep))
leap += keeps_leap(rep)
print("{:<14}{:>10.3f}{:>11.3f}{:>11.3f}".format(
name, statistics.fmean(cov), statistics.fmean(taus),
leap / N))
print("textbook order tau(real) = {:.3f}".format(tau(sorted(REAL))))
main()Single source and citation
Fan, Y. (2026). Reasoning-Trajectory Profiling, Process-Tracing Games and Separating Cognitive Structure from Stated Intentions and Outcome Luck (Version 1.0.0), Chapter 1: The Limits of Self-Report: After-the-Fact Reasons, Thinking Aloud and Reactivity. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-385 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026385,
author = {Fan, Yafu},
title = {Reasoning-Trajectory Profiling, Process-Tracing Games and Separating Cognitive Structure from Stated Intentions and Outcome Luck},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 2 · Four Process-Tracing Tests Written as Likelihood Ratios
- Chapter 3 · The Reasoning Graph: Nodes, Leaps, Backtracks and Revisions
- Chapter 4 · Fingerprints under Pressure: Intraclass Correlation and Rater Agreement
- Chapter 5 · The Cost of Forgery: Probing Depth, Memory Load and Bounds on Detection
- Chapter 6 · Stripping Out Outcome Luck: Track Records, Sequential Tests and Multi-Method Composites
- Chapter 7 · Classical Sources: the Analects on Seeing, Watching and Examining; Bayes and Aristotle on Choice
- Chapter 8 · Field Tools and Checker Specifications: the Reasoning-Trajectory Selection Ledger, the Scorer, Pre-Registration and Refutation