What does Calibration and Collective Intelligence conclude, in numbers, about “Signal Detection Theory: Seeing Clearly and Where to Set the Threshold”?
Answer: The first three judges have almost the same sensitivity (\(d'\) about 2); the difference lies entirely in the threshold. The neutral judge has \(c = 0\) and \(\beta = 1\); the liberal one has \(c = -0.70\) and a hit rate of 95 %, paid for with 40 % false alarms; the conservative one has \(c = +1.00\), almost never raises a false alarm (2.3 %) and misses half the signals. Judged by hit rate alone, the liberal judge looks "best"; judged by false alarms alone, the conservative one does; in fact all three see equally well. The fourth judge has a 90 % hit rate, which looks good, but a 69 % false-alarm rate and \(d'\) of only 0.79: he is mostly saying "yes" to everything.
Scope of the numbers: This book is not counselling, medical or investment advice. Marked boxes separate published evidence, runnable Python calculations (16 programs, all checked), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python onlyprinted values that match the answer: -0.70, 0.79, 1.00
| observer | H FA | d' c beta | |
| neutral | 0.840 0.160 | | 1.99 +0.00 1.00 |
| liberal (says yes a lot) | 0.950 0.400 | | 1.90 -0.70 0.27 |
| conservative | 0.500 0.023 | | 2.00 +1.00 7.32 |
| weak but liberal | 0.900 0.690 | | 0.79 -0.89 0.50 |
| ROC d' | 0.5: AUC = 0.638; (FA, H) at criteria -1, 0, 1, 2: |
| (0.84, 0.93) (0.50, 0.69) (0.16, 0.31) (0.02, 0.07) | |
| ROC d' | 1.0: AUC = 0.760; (FA, H) at criteria -1, 0, 1, 2: |
| (0.84, 0.98) (0.50, 0.84) (0.16, 0.50) (0.02, 0.16) | |
| ROC d' | 2.0: AUC = 0.921; (FA, H) at criteria -1, 0, 1, 2: |
| (0.84, 1.00) (0.50, 0.98) (0.16, 0.84) (0.02, 0.50) |
Key formula
Show the full program
# Equal-variance Gaussian signal detection: noise N(0, 1), signal N(d', 1).
# d' = z(H) - z(FA), c = -(z(H) + z(FA)) / 2, beta = exp(c d'), AUC = Phi(d' / sqrt 2).
import math
from statistics import NormalDist
N = NormalDist()
def sdt(h, fa):
zh, zf = N.inv_cdf(h), N.inv_cdf(fa)
d = zh - zf
c = -(zh + zf) / 2 + 0.0
return d, c, math.exp(c * d)
def roc(d, crits):
return [(1 - N.cdf(k), 1 - N.cdf(k - d)) for k in crits]
if __name__ == "__main__":
print("observer | H FA | d' c beta")
for name, h, fa in (("neutral", 0.84, 0.16), ("liberal (says yes a lot)", 0.95, 0.40),
("conservative", 0.50, 0.023), ("weak but liberal", 0.90, 0.69)):
d, c, b = sdt(h, fa)
print(f"{name:24s} | {h:.3f} {fa:.3f} | {d:5.2f} {c:+6.2f} {b:6.2f}")
for d in (0.5, 1.0, 2.0):
pts = roc(d, (-1.0, 0.0, 1.0, 2.0))
auc = N.cdf(d / math.sqrt(2))
print(f"ROC d' = {d}: AUC = {auc:.3f}; (FA, H) at criteria -1, 0, 1, 2:")
print(" " + " ".join(f"({f:.2f}, {h:.2f})" for f, h in pts))Single source and citation
Fan, Y. (2026). Probabilistic Judgement, Brier-Score Calibration and the Statistics of Collective Intelligence (Version 1.0.0), Chapter 4: Signal Detection Theory: Seeing Clearly and Where to Set the Threshold. 5D Library. https://doi.org/10.5281/zenodo.23764998
DOI: 10.5281/zenodo.23764998 · Read the monograph · 5DL-B-026 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026026,
author = {Fan, Yafu},
title = {Probabilistic Judgement, Brier-Score Calibration and the Statistics of Collective Intelligence},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
doi = {10.5281/zenodo.23764998},
url = {https://doi.org/10.5281/zenodo.23764998}
}Other chapter answers in this book
- Chapter 1 · Tetlock's Forecasting Research: Experts, Foxes and Superforecasters
- Chapter 2 · Strictly Proper Scoring Rules: The Brier Score and Murphy's Decomposition
- Chapter 3 · Base-Rate Neglect and Natural Frequencies
- Chapter 5 · Reference-Class Forecasting and the Planning Fallacy
- Chapter 6 · The Statistics of Collective Intelligence
- Chapter 8 · Practice and a Pre-registered Trial