Two different abilities

A radar operator watches a blip on the screen: is it an aircraft or a burst of noise? A doctor looks at a scan: is it a lesion or normal variation? An investor sees a piece of news: is it an opportunity or noise? In each case the judgement involves two things. First, how clearly one sees: how far apart signal and noise lie in the evidence. Second, where one sets the threshold: how strong the evidence must be before one says “yes”. A judge can have excellent eyesight but too low a threshold, or ordinary eyesight and a well-placed threshold.

Signal detection theory separates the two (Green and Swets 1966; Macmillan and Creelman 2005). It grew out of radar and psychophysics research after the Second World War and was later applied to medical diagnosis, weather forecasting, lie detection, memory research and many other fields (Swets, Dawes, and Monahan 2000).

Model and measures

The simplest version is the equal-variance Gaussian model: on “noise” trials without a signal, the evidence $x$ the judge experiences follows $N(0, 1)$; on signal trials it follows $N(d', 1)$. The judge says “signal” when $x$ exceeds a criterion $k$. There are four outcomes: hit (signal present, “yes”), miss, false alarm (no signal, “yes”) and correct rejection. With hit rate $H$, false-alarm rate $FA$ and $z(\cdot)$ the inverse of the standard normal distribution function,

$$d' = z(H) – z(FA), \qquad c = -\frac{z(H) + z(FA)}{2}, \qquad \beta = e^{c\,d'}.$$

$d'$ is sensitivity, the number of standard deviations between the means of the two distributions; $c$ is the position of the criterion, with $c = 0$ meaning the threshold lies midway between the two distributions, $c < 0$ a bias towards “yes” (liberal) and $c > 0$ towards “no” (conservative); $\beta$ is the likelihood ratio $P(x \mid S)/P(x \mid N)$ at the threshold.

# Equal-variance Gaussian signal detection: noise N(0, 1), signal N(d', 1).
# d' = z(H) - z(FA), c = -(z(H) + z(FA)) / 2, beta = exp(c d'), AUC = Phi(d' / sqrt 2).
import math
from statistics import NormalDist

N = NormalDist()

def sdt(h, fa):
    zh, zf = N.inv_cdf(h), N.inv_cdf(fa)
    d = zh - zf
    c = -(zh + zf) / 2 + 0.0
    return d, c, math.exp(c * d)

def roc(d, crits):
    return [(1 - N.cdf(k), 1 - N.cdf(k - d)) for k in crits]

if __name__ == "__main__":
    print("observer                 |   H     FA  |   d'      c    beta")
    for name, h, fa in (("neutral", 0.84, 0.16), ("liberal (says yes a lot)", 0.95, 0.40),
                        ("conservative", 0.50, 0.023), ("weak but liberal", 0.90, 0.69)):
        d, c, b = sdt(h, fa)
        print(f"{name:24s} | {h:.3f} {fa:.3f} | {d:5.2f} {c:+6.2f} {b:6.2f}")
    for d in (0.5, 1.0, 2.0):
        pts = roc(d, (-1.0, 0.0, 1.0, 2.0))
        auc = N.cdf(d / math.sqrt(2))
        print(f"ROC d' = {d}: AUC = {auc:.3f}; (FA, H) at criteria -1, 0, 1, 2:")
        print("    " + "  ".join(f"({f:.2f}, {h:.2f})" for f, h in pts))
# observer                 |   H     FA  |   d'      c    beta
# neutral                  | 0.840 0.160 |  1.99  +0.00   1.00
# liberal (says yes a lot) | 0.950 0.400 |  1.90  -0.70   0.27
# conservative             | 0.500 0.023 |  2.00  +1.00   7.32
# weak but liberal         | 0.900 0.690 |  0.79  -0.89   0.50
# ROC d' = 0.5: AUC = 0.638; (FA, H) at criteria -1, 0, 1, 2:
#     (0.84, 0.93)  (0.50, 0.69)  (0.16, 0.31)  (0.02, 0.07)
# ROC d' = 1.0: AUC = 0.760; (FA, H) at criteria -1, 0, 1, 2:
#     (0.84, 0.98)  (0.50, 0.84)  (0.16, 0.50)  (0.02, 0.16)
# ROC d' = 2.0: AUC = 0.921; (FA, H) at criteria -1, 0, 1, 2:
#     (0.84, 1.00)  (0.50, 0.98)  (0.16, 0.84)  (0.02, 0.50)

The first three judges have almost the same sensitivity ($d'$ about 2); the difference lies entirely in the threshold. The neutral judge has $c = 0$ and $\beta = 1$; the liberal one has $c = -0.70$ and a hit rate of 95 %, paid for with 40 % false alarms; the conservative one has $c = +1.00$, almost never raises a false alarm (2.3 %) and misses half the signals. Judged by hit rate alone, the liberal judge looks “best”; judged by false alarms alone, the conservative one does; in fact all three see equally well. The fourth judge has a 90 % hit rate, which looks good, but a 69 % false-alarm rate and $d'$ of only 0.79: he is mostly saying “yes” to everything.

The optimal threshold

Where should the threshold be? That depends on two things: how common the signal is to begin with, and what each of the four outcomes gains or costs.

# Optimal criterion (Green & Swets 1966): beta* = [P(N)/P(S)] [(V_CR + C_FA)/(V_H +
# C_M)]; on the evidence axis x* = ln(beta*)/d' + d'/2. d' = 1.5, P(S) = 0.10;
# illustrative payoffs.
import math
from statistics import NormalDist

N = NormalDist()

def expected_value(x, d, ps, vh, cm, vcr, cfa):
    h, fa = 1 - N.cdf(x - d), 1 - N.cdf(x)
    return ps * (h * vh - (1 - h) * cm) + (1 - ps) * ((1 - fa) * vcr - fa * cfa), h, fa

def report(name, d, ps, vh, cm, vcr, cfa):
    beta = (1 - ps) / ps * (vcr + cfa) / (vh + cm)
    xs = math.log(beta) / d + d / 2
    grid = [i / 100 for i in range(-400, 601)]
    xg = max(grid, key=lambda x: expected_value(x, d, ps, vh, cm, vcr, cfa)[0])
    print(f"{name}: beta* = {beta:.2f}, criterion x* = {xs:+.2f} (grid search {xg:+.2f})")
    for lab, x in (("optimal x*", xs), ("neutral beta = 1", d / 2),
                   ("act on everything", -9.0), ("act on nothing", 9.0)):
        ev, h, fa = expected_value(x, d, ps, vh, cm, vcr, cfa)
        print(f"   {lab:18s} H = {h:.2f}  FA = {fa:.3f}  expected value {ev:+.3f}")

if __name__ == "__main__":
    report("opportunity", 1.5, 0.10, vh=5, cm=1, vcr=0, cfa=4)
    report("screening  ", 1.5, 0.10, vh=0, cm=20, vcr=0, cfa=1)
# opportunity: beta* = 6.00, criterion x* = +1.94 (grid search +1.94)
#    optimal x*         H = 0.33  FA = 0.026  expected value +0.004
#    neutral beta = 1   H = 0.77  FA = 0.227  expected value -0.452
#    act on everything  H = 1.00  FA = 1.000  expected value -3.100
#    act on nothing     H = 0.00  FA = 0.000  expected value -0.100
# screening  : beta* = 0.45, criterion x* = +0.22 (grid search +0.22)
#    optimal x*         H = 0.90  FA = 0.414  expected value -0.572
#    neutral beta = 1   H = 0.77  FA = 0.227  expected value -0.657
#    act on everything  H = 1.00  FA = 1.000  expected value -0.900
#    act on nothing     H = 0.00  FA = 0.000  expected value -2.000

In the “opportunity” setting, $\beta^* = 9 \times 4/6 = 6.0$ and the threshold must be high: the optimal strategy seizes only a third of real opportunities and holds false alarms to 2.6 %, for an expected value of about +0.004 per case, barely positive. With a “neutral” threshold ($\beta = 1$) one seizes 77 % of real opportunities but also acts on 22.7 % of false ones, for an expected value of −0.452 per case; acting on everything gives −3.100, far worse than doing nothing (−0.100). When real opportunities are rare and false ones are costly, most things that look like opportunities should be passed over, even if your sight is good.

The “screening” setting is the opposite: $\beta^* = 9 \times 1/20 = 0.45$, the threshold goes low, the hit rate reaches 90 %, and the price is 41.4 % false alarms. That is reasonable in screening, because a false alarm means only one more check. With the same sensitivity, the optimal hit rate is 33 % in one setting and 90 % in the other: there is no absolutely “right” threshold, only a threshold that is right relative to the base rate and the costs.

An analogy: “always spotting an opportunity” and “always being targeted”

Some people always feel they have spotted an opportunity: every piece of news looks like a signal, every project like the next big thing. In this chapter’s terms, the threshold may be too low ($c$ far below 0), or the base rate of opportunities $P(S)$ may be overestimated, or the cost of a false alarm $C_{FA}$ underestimated. The program above shows that where signals are rare, such a threshold brings many false alarms, and even a high hit rate can leave the overall result poor.

Others always feel targeted: a colleague’s remark, a friend’s late reply, are read as signals of hostility. In the model this too appears as a low threshold for the “hostility” signal. But here special care is needed. First, the model assumes there is an objective “signal” and “noise”, whereas in interpersonal situations the other person’s intentions are often genuinely ambiguous, and bystanders cannot always tell either. Second, some people really are in environments where they are targeted, and then a low threshold is reasonable vigilance, not bias. Third, the model cannot distinguish ordinary suspiciousness from situations that need professional help. So the analogy can at most prompt a question (“in my situation, how common are such events in the first place, and what do false alarms and misses cost?”); it cannot provide the answer.

Chapter summary

Signal detection theory splits a judgement into sensitivity $d'$ and threshold $c$ (or likelihood ratio $\beta$): the same sight can produce completely different hit and false-alarm rates depending on the threshold. The optimal threshold $\beta^* = [P(N)/P(S)] \cdot [(V_{CR} + C_{FA})/(V_H + C_M)]$ is set jointly by the base rate of signals and the gains and costs. When signals are rare and false alarms costly the threshold should be high; when misses are costly it should be low. Applying this framework to “always spotting an opportunity” or “always being targeted” is only an analogy; if the latter persists and causes distress, professional help should be sought. The next chapter discusses another kind of base rate: the history of similar projects.