What does Reasoning Trajectories and Outcome Luck conclude, in numbers, about “The Limits of Self-Report: After-the-Fact Reasons, Thinking Aloud and Reactivity”?

Answer: Thinking aloud keeps 0.919 of the steps, its Kendall correlation with the real order is 0.977, and 0.968 of reports still show at least one leap. Retrospective recall keeps only 0.648 of the steps, its correlation falls to 0.833, and only 0.235 of reports still show a leap.

Scope of the numbers: After-the-fact explanations are often folk theories: in a synthetic experiment, retrospective self-reports correlate only -0.013 with the weights used but 0.826 with an observer's guess; concurrent thinking aloud keeps order and leaps. Parameters are illustrative and all calculations come from the book's programs.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: 0.235, 0.648, 0.833, 0.919, 0.968, 0.977

synthetic people per condition: 2,000
conditioncoveragetau(real)leap kept
concurrent0.9190.9770.968
retrospective0.6480.8330.235
textbook order tau(real) = 0.786
Show the full program
# Chapter 1. p12: does the report keep the order and the leaps?
# A reasoning episode has eight steps 0..7. The textbook order is 0..7;
# the real episode jumped twice: 4 came before 2 and 3, and 6 before 5.
# Concurrent think-aloud: each step reported with prob 0.92, neighbours
# swapped with prob 0.04. Retrospective account: each step recalled with
# prob 0.65; with prob 0.6 the person rebuilds the textbook order,
# otherwise keeps the real order with neighbour swaps at 0.10.
# Measures over 2,000 synthetic people per condition: coverage, Kendall
# tau against the real order, and the share of reports that still show
# a leap (4 before 2, or 6 before 5). Parameters are illustrative.
import random
import statistics

REAL = [0, 1, 4, 2, 3, 6, 5, 7]
N = 2000


def swaps(seq, p, rng):
    s = list(seq)
    for i in range(len(s) - 1):
        if rng.random() < p:
            s[i], s[i + 1] = s[i + 1], s[i]
    return s


def tau(rep):
    pos = {v: i for i, v in enumerate(REAL)}
    conc = disc = 0
    for i in range(len(rep)):
        for j in range(i + 1, len(rep)):
            if pos[rep[i]] < pos[rep[j]]:
                conc += 1
            else:
                disc += 1
    tot = conc + disc
    return (conc - disc) / tot if tot else 0.0


def keeps_leap(rep):
    where = {v: i for i, v in enumerate(rep)}
    for a, b in ((4, 2), (6, 5)):
        if a in where and b in where and where[a] < where[b]:
            return 1
    return 0


def concurrent(rng):
    kept = [s for s in REAL if rng.random() < 0.92]
    return swaps(kept, 0.04, rng)


def retrospective(rng):
    kept = [s for s in REAL if rng.random() < 0.65]
    if rng.random() < 0.6:
        return sorted(kept)
    return swaps(kept, 0.10, rng)


def main():
    rng = random.Random(38512)
    print("synthetic people per condition: {:,}".format(N))
    print("condition       coverage  tau(real)  leap kept")
    for name, fn in (("concurrent", concurrent),
                     ("retrospective", retrospective)):
        cov, taus, leap = [], [], 0
        for _ in range(N):
            rep = fn(rng)
            cov.append(len(rep) / len(REAL))
            taus.append(tau(rep))
            leap += keeps_leap(rep)
        print("{:<14}{:>10.3f}{:>11.3f}{:>11.3f}".format(
            name, statistics.fmean(cov), statistics.fmean(taus),
            leap / N))
    print("textbook order tau(real) = {:.3f}".format(tau(sorted(REAL))))


main()

Single source and citation

Fan, Y. (2026). Reasoning-Trajectory Profiling, Process-Tracing Games and Separating Cognitive Structure from Stated Intentions and Outcome Luck (Version 1.0.0), Chapter 1: The Limits of Self-Report: After-the-Fact Reasons, Thinking Aloud and Reactivity. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-385 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026385,
  author    = {Fan, Yafu},
  title     = {Reasoning-Trajectory Profiling, Process-Tracing Games and Separating Cognitive Structure from Stated Intentions and Outcome Luck},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book