Not an oracle

A language model is trained to predict text. Given a sequence of tokens (pieces of words), it assigns a probability to every possible next token, $p_\theta(y_t \mid y_{<t})$, and a reply is produced by choosing tokens one after another from these distributions. The parameters $\theta$ are fitted to large collections of text, mostly written by people, and then adjusted by further training on examples and on human or automated preferences. To a first approximation, then, a model is a learned conditional distribution: what it says depends on what it is given.

This picture is incomplete in two important ways. First, the model is not the distribution of its training text; it is a compressed, generalising approximation to it, which can produce sentences that never appeared in any document and can state falsehoods fluently (Ji et al. 2023). Second, the fine-tuning that makes models helpful also reshapes their outputs.

Information in the question

Information theory gives exact words for what a question contributes (Shannon 1948; Cover and Thomas 2006). The entropy $H(Y)$ of the answers measures how uncertain they are before we see the question. The conditional entropy $H(Y \mid X)$ measures the uncertainty that remains once the question $X$ is known. The difference is the mutual information,

$$I(X;Y) = H(Y) – H(Y \mid X),$$

the number of bits by which, on average, knowing the question narrows the answer.

import math

def H(p):
    return -sum(x * math.log2(x) for x in p if x > 0)

# A toy "corpus": how often each of six continuations follows two kinds of prompt.
# Columns: generic summary, cliché, textbook example, cross-field analogy, novel hypothesis, testable protocol
counts = {
    "vague prompt":       [400, 300, 200, 50, 30, 20],
    "constrained prompt": [40, 10, 60, 250, 300, 340],
}
p_x = {"vague prompt": 0.5, "constrained prompt": 0.5}
p_y_given_x = {x: [c / sum(cs) for c in cs] for x, cs in counts.items()}
p_y = [sum(p_x[x] * p_y_given_x[x][j] for x in p_x) for j in range(6)]

H_y = H(p_y)
H_y_x = sum(p_x[x] * H(p_y_given_x[x]) for x in p_x)
print(f"H(Y) = {H_y:.3f} bits, H(Y|X) = {H_y_x:.3f} bits, I(X;Y) = {H_y - H_y_x:.3f} bits")
for x, p in p_y_given_x.items():
    top = max(range(6), key=lambda j: p[j])
    print(f"{x:18s} H = {H(p):.2f} bits, most likely continuation #{top + 1} (p = {p[top]:.2f}), "
          f"share on #4–6 = {sum(p[3:]):.2f}")
# H(Y) = 2.565 bits, H(Y|X) = 2.020 bits, I(X;Y) = 0.544 bits
# vague prompt       H = 1.99 bits, most likely continuation #1 (p = 0.40), share on #4–6 = 0.10
# constrained prompt H = 2.05 bits, most likely continuation #6 (p = 0.34), share on #4–6 = 0.89

The result is instructive in a way that is easy to miss. The constrained prompt does not make the answer much less uncertain: its entropy is about the same as the vague prompt’s. What it changes is where the probability lies. After the vague prompt, 90 % of the mass is on the generic summary, the cliché and the textbook example; after the constrained prompt, 89 % is on the analogy, the hypothesis and the protocol. The information in a question is less a narrowing than a redirection.

What decoding can and cannot do

Systems usually let users, or their designers, set a temperature that sharpens or flattens the next-token distribution. It is tempting to think that a higher temperature produces more original output.

import math

# Next-token choice from a Zipf-shaped distribution over 1,000 candidates.
# Greedy decoding always returns rank 1; sampling at temperature T reshapes p ∝ p^(1/T).
N = 1000
base = [1 / (r + 1) ** 1.1 for r in range(N)]
Z = sum(base)
p = [x / Z for x in base]

def reshape(p, T):
    q = [x ** (1 / T) for x in p]
    s = sum(q)
    return [x / s for x in q]

for T in (0.3, 0.7, 1.0, 1.3):
    q = reshape(p, T)
    print(f"T = {T}: P(rank 1) = {q[0]:.2f}, P(top 10) = {sum(q[:10]):.2f}, P(rank > 100) = {sum(q[100:]):.2f}")
print("Temperature spreads probability, but it spreads it over whatever the context made likely;")
print("it cannot put mass on continuations the context did not support.")
# T = 0.3: P(rank 1) = 0.90, P(top 10) = 1.00, P(rank > 100) = 0.00
# T = 0.7: P(rank 1) = 0.43, P(top 10) = 0.82, P(rank > 100) = 0.04
# T = 1.0: P(rank 1) = 0.18, P(top 10) = 0.48, P(rank > 100) = 0.23
# T = 1.3: P(rank 1) = 0.08, P(top 10) = 0.26, P(rank > 100) = 0.44
# Temperature spreads probability, but it spreads it over whatever the context made likely;
# it cannot put mass on continuations the context did not support.

Low temperature concentrates almost everything on the most likely token; high temperature spreads probability down the ranking. But it spreads it in proportion to what the context has already made likely. Temperature adds variety; it does not add direction. A low-information prompt answered at high temperature yields varied versions of the typical answer, and sometimes nonsense, not the answer a well-framed question would have reached.