What does Multi-Model Voting with Correlated Errors conclude, in numbers, about “Measuring Correlation: Pairwise Correlation, the Q Statistic, Effective Votes and Colliding Wrong Answers”?
Answer: Correlation can be measured, and it should be. The phi coefficient and the moment estimate correspond directly to \(\rho\) in the model; the Q statistic is more sensitive but runs on a different scale; the both-wrong share is the most intuitive. Once \(\rho\) is known, the cap \(1/\rho\) on effective votes tells the owner how much independent information remains to be bought, and multiple-choice reviews need one more measurement, the collision rate. The next chapter turns to action: since correlation comes from common causes, which methods can break it, and how much does each break?
Scope of the numbers: Its parameters are illustrative; all calculations are generated by its programs; it is general business analysis, not business advice. Marked boxes separate published evidence, runnable Python calculations (19 programs), classical sources and the author's hypotheses.
Data table
The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python only
| 600 items x 5 models, true mean accuracy 0.75 | |
| true rho p-hat phi Q both-wrong (1-p)^2 | |
| 0.0 | 0.7490 -0.0105 -0.0305 0.0610 0.0630 |
| 0.25 | 0.7630 0.2649 0.5747 0.1040 0.0562 |
| moment estimate of rho and the majority it predicts | |
| true rho rho-hat predicted maj observed maj | |
| 0.0 | -0.0102 0.8954 0.8950 |
| 0.25 | 0.2654 0.8196 0.8183 |
Show the full program
# Chapter 3. p31: measuring error correlation from a labelled sample.
# Model: a synthetic item-by-model table of right (1) / wrong (0).
# Each of 600 items draws a shared difficulty theta ~ Beta(a, b) with
# mean 0.75 and correlation rho; five models answer it independently
# given theta (fixed seed). A second table has rho = 0. Estimators:
# pairwise phi (Pearson on 0/1), the Q statistic of Yule used by
# Kuncheva and Whitaker, the both-wrong rate against (1 - p)^2, and a
# moment estimate of rho from the spread of right answers per item,
# rho = (s^2 / (n p (1 - p)) - 1) / (n - 1). The estimate is then fed
# to the Beta-binomial majority of p21. All parameters illustrative.
import itertools
import math
import random
def lbeta(a, b):
return math.lgamma(a) + math.lgamma(b) - math.lgamma(a + b)
def ab(p, rho):
s = 1 / rho - 1
return p * s, (1 - p) * s
def bb_major(n, p, rho):
if rho == 0:
lp, lq = math.log(p), math.log(1 - p)
terms = [math.exp(math.lgamma(n + 1) - math.lgamma(k + 1)
- math.lgamma(n - k + 1) + k * lp
+ (n - k) * lq)
for k in range(n // 2 + 1, n + 1)]
return math.fsum(terms)
a, b = ab(p, rho)
terms = [math.exp(math.lgamma(n + 1) - math.lgamma(k + 1)
- math.lgamma(n - k + 1) + lbeta(k + a, n - k + b)
- lbeta(a, b))
for k in range(n // 2 + 1, n + 1)]
return math.fsum(terms)
def betacf(a, b, x):
tiny = 1e-300
c, d = 1.0, 1.0 - (a + b) * x / (a + 1)
d = 1.0 / (d if abs(d) > tiny else tiny)
h = d
for m in range(1, 400):
m2 = 2 * m
for num in (m * (b - m) * x / ((a + m2 - 1) * (a + m2)),
-(a + m) * (a + b + m) * x / ((a + m2) * (a + m2 + 1))):
d = 1.0 + num * d
d = 1.0 / (d if abs(d) > tiny else tiny)
c = 1.0 + num / c
c = c if abs(c) > tiny else tiny
h *= d * c
if abs(d * c - 1.0) < 1e-15:
break
return h
def betai(a, b, x):
if x <= 0:
return 0.0
if x >= 1:
return 1.0
lead = math.exp(a * math.log(x) + b * math.log(1 - x) - lbeta(a, b))
if x < (a + 1) / (a + b + 2):
return lead * betacf(a, b, x) / a
return 1.0 - lead * betacf(b, a, 1 - x) / b
def limit(p, rho):
if rho == 0:
return 1.0 if p > 0.5 else 0.0
a, b = ab(p, rho)
return 1.0 - betai(a, b, 0.5)
def table(rng, items, n, p, rho):
rows = []
for _ in range(items):
if rho > 0:
a, b = ab(p, rho)
th = rng.betavariate(a, b)
else:
th = p
rows.append([1 if rng.random() < th else 0 for _ in range(n)])
return rows
def stats(rows):
n = len(rows[0])
m = len(rows)
ph = math.fsum(math.fsum(r) for r in rows) / (m * n)
phis, qs, bw = [], [], []
for i, j in itertools.combinations(range(n), 2):
c = [[0, 0], [0, 0]]
for r in rows:
c[r[i]][r[j]] += 1
n11, n00, n10, n01 = c[1][1], c[0][0], c[1][0], c[0][1]
den = math.sqrt((n11 + n10) * (n01 + n00)
* (n11 + n01) * (n10 + n00))
phis.append((n11 * n00 - n01 * n10) / den)
qs.append((n11 * n00 - n01 * n10) / (n11 * n00 + n01 * n10))
bw.append(n00 / m)
ks = [math.fsum(r) for r in rows]
kbar = math.fsum(ks) / m
s2 = math.fsum((k - kbar) ** 2 for k in ks) / (m - 1)
rho = (s2 / (n * ph * (1 - ph)) - 1) / (n - 1)
maj = math.fsum(1.0 for k in ks if k > n / 2) / m
return ph, math.fsum(phis) / len(phis), math.fsum(qs) / len(qs), \
math.fsum(bw) / len(bw), rho, maj
if __name__ == "__main__":
rng = random.Random(20261113)
print("600 items x 5 models, true mean accuracy 0.75")
print(" true rho p-hat phi Q both-wrong (1-p)^2")
res = {}
for rho in (0.0, 0.25):
ph, phi, q, bw, rh, maj = stats(table(rng, 600, 5, 0.75, rho))
res[rho] = (ph, rh, maj)
print(" {:<8} {:.4f} {:.4f} {:.4f} {:.4f} {:.4f}".format(
rho, ph, phi, q, bw, (1 - ph) ** 2))
print("moment estimate of rho and the majority it predicts")
print(" true rho rho-hat predicted maj observed maj")
for rho in (0.0, 0.25):
ph, rh, maj = res[rho]
pred = bb_major(5, ph, max(rh, 0.0))
print(" {:<8} {:7.4f} {:13.4f} {:12.4f}".format(
rho, rh, pred, maj))Single source and citation
Fan, Y. (2026). Condorcet's Jury Theorem with Correlated Errors: When Multi-Model Voting Gets More Accurate and When Shared Origins Make Everyone Wrong Together (Version 1.0.0), Chapter 3: Measuring Correlation: Pairwise Correlation, the Q Statistic, Effective Votes and Colliding Wrong Answers. 5D Library. DOI pending
DOI pending · monograph page goes live after publication · 5DL-B-309 · v1.0.0 · CC-BY-NC-ND-4.0
BibTeX
@book{fan2026309,
author = {Fan, Yafu},
title = {Condorcet's Jury Theorem with Correlated Errors: When Multi-Model Voting Gets More Accurate and When Shared Origins Make Everyone Wrong Together},
year = {2026},
publisher = {5D Library},
edition = {1.0.0},
note = {DOI pending}
}Other chapter answers in this book
- Chapter 1 · Condorcet's Binomial Majority: Why Independent Votes Grow More Accurate
- Chapter 2 · Correlated Errors and the Ceiling: the Beta-Binomial Model, Opinion Leaders and Shared Origins
- Chapter 4 · Comparing Ways to Decorrelate: Resampling, Different Prompts, Different Models and Standard-Library Checks
- Chapter 5 · Weighted Majorities: Log-Odds Weights and the Double Counting of Correlated Clusters
- Chapter 6 · How Many Models and How Many Labelled Questions: the Vote Ledger and the Labelling Ledger below the Ceiling
- Chapter 7 · Classical Sources East and West: "When Three Read, Follow the Two" in the Great Plan and Condorcet's Essai of 1785
- Chapter 8 · Closing the Loop on Decorrelated Multi-Model Verdicts: a Procedure, the Correlated Voting Accuracy Calculator and a Pre-Registered Plan of Refutation