What does Multi-Model Voting with Correlated Errors conclude, in numbers, about “Measuring Correlation: Pairwise Correlation, the Q Statistic, Effective Votes and Colliding Wrong Answers”?

Answer: Correlation can be measured, and it should be. The phi coefficient and the moment estimate correspond directly to \(\rho\) in the model; the Q statistic is more sensitive but runs on a different scale; the both-wrong share is the most intuitive. Once \(\rho\) is known, the cap \(1/\rho\) on effective votes tells the owner how much independent information remains to be bought, and multiple-choice reviews need one more measurement, the collision rate. The next chapter turns to action: since correlation comes from common causes, which methods can break it, and how much does each break?

Scope of the numbers: Its parameters are illustrative; all calculations are generated by its programs; it is general business analysis, not business advice. Marked boxes separate published evidence, runnable Python calculations (19 programs), classical sources and the author's hypotheses.

中文版

Data table

The numbers below are printed verbatim by the chapter's Python program and can be reproduced by running it. standard-library Python only

600 items x 5 models, true mean accuracy 0.75
true rho p-hat phi Q both-wrong (1-p)^2
0.00.7490 -0.0105 -0.0305 0.0610 0.0630
0.250.7630 0.2649 0.5747 0.1040 0.0562
moment estimate of rho and the majority it predicts
true rho rho-hat predicted maj observed maj
0.0-0.0102 0.8954 0.8950
0.250.2654 0.8196 0.8183
Show the full program
# Chapter 3. p31: measuring error correlation from a labelled sample.
# Model: a synthetic item-by-model table of right (1) / wrong (0).
# Each of 600 items draws a shared difficulty theta ~ Beta(a, b) with
# mean 0.75 and correlation rho; five models answer it independently
# given theta (fixed seed). A second table has rho = 0. Estimators:
# pairwise phi (Pearson on 0/1), the Q statistic of Yule used by
# Kuncheva and Whitaker, the both-wrong rate against (1 - p)^2, and a
# moment estimate of rho from the spread of right answers per item,
# rho = (s^2 / (n p (1 - p)) - 1) / (n - 1). The estimate is then fed
# to the Beta-binomial majority of p21. All parameters illustrative.
import itertools
import math
import random



def lbeta(a, b):
    return math.lgamma(a) + math.lgamma(b) - math.lgamma(a + b)


def ab(p, rho):
    s = 1 / rho - 1
    return p * s, (1 - p) * s


def bb_major(n, p, rho):
    if rho == 0:
        lp, lq = math.log(p), math.log(1 - p)
        terms = [math.exp(math.lgamma(n + 1) - math.lgamma(k + 1)
                          - math.lgamma(n - k + 1) + k * lp
                          + (n - k) * lq)
                 for k in range(n // 2 + 1, n + 1)]
        return math.fsum(terms)
    a, b = ab(p, rho)
    terms = [math.exp(math.lgamma(n + 1) - math.lgamma(k + 1)
                      - math.lgamma(n - k + 1) + lbeta(k + a, n - k + b)
                      - lbeta(a, b))
             for k in range(n // 2 + 1, n + 1)]
    return math.fsum(terms)


def betacf(a, b, x):
    tiny = 1e-300
    c, d = 1.0, 1.0 - (a + b) * x / (a + 1)
    d = 1.0 / (d if abs(d) > tiny else tiny)
    h = d
    for m in range(1, 400):
        m2 = 2 * m
        for num in (m * (b - m) * x / ((a + m2 - 1) * (a + m2)),
                    -(a + m) * (a + b + m) * x / ((a + m2) * (a + m2 + 1))):
            d = 1.0 + num * d
            d = 1.0 / (d if abs(d) > tiny else tiny)
            c = 1.0 + num / c
            c = c if abs(c) > tiny else tiny
            h *= d * c
        if abs(d * c - 1.0) < 1e-15:
            break
    return h


def betai(a, b, x):
    if x <= 0:
        return 0.0
    if x >= 1:
        return 1.0
    lead = math.exp(a * math.log(x) + b * math.log(1 - x) - lbeta(a, b))
    if x < (a + 1) / (a + b + 2):
        return lead * betacf(a, b, x) / a
    return 1.0 - lead * betacf(b, a, 1 - x) / b


def limit(p, rho):
    if rho == 0:
        return 1.0 if p > 0.5 else 0.0
    a, b = ab(p, rho)
    return 1.0 - betai(a, b, 0.5)


def table(rng, items, n, p, rho):
    rows = []
    for _ in range(items):
        if rho > 0:
            a, b = ab(p, rho)
            th = rng.betavariate(a, b)
        else:
            th = p
        rows.append([1 if rng.random() < th else 0 for _ in range(n)])
    return rows


def stats(rows):
    n = len(rows[0])
    m = len(rows)
    ph = math.fsum(math.fsum(r) for r in rows) / (m * n)
    phis, qs, bw = [], [], []
    for i, j in itertools.combinations(range(n), 2):
        c = [[0, 0], [0, 0]]
        for r in rows:
            c[r[i]][r[j]] += 1
        n11, n00, n10, n01 = c[1][1], c[0][0], c[1][0], c[0][1]
        den = math.sqrt((n11 + n10) * (n01 + n00)
                        * (n11 + n01) * (n10 + n00))
        phis.append((n11 * n00 - n01 * n10) / den)
        qs.append((n11 * n00 - n01 * n10) / (n11 * n00 + n01 * n10))
        bw.append(n00 / m)
    ks = [math.fsum(r) for r in rows]
    kbar = math.fsum(ks) / m
    s2 = math.fsum((k - kbar) ** 2 for k in ks) / (m - 1)
    rho = (s2 / (n * ph * (1 - ph)) - 1) / (n - 1)
    maj = math.fsum(1.0 for k in ks if k > n / 2) / m
    return ph, math.fsum(phis) / len(phis), math.fsum(qs) / len(qs), \
        math.fsum(bw) / len(bw), rho, maj


if __name__ == "__main__":
    rng = random.Random(20261113)
    print("600 items x 5 models, true mean accuracy 0.75")
    print("  true rho   p-hat    phi      Q     both-wrong (1-p)^2")
    res = {}
    for rho in (0.0, 0.25):
        ph, phi, q, bw, rh, maj = stats(table(rng, 600, 5, 0.75, rho))
        res[rho] = (ph, rh, maj)
        print("  {:<8} {:.4f}  {:.4f}  {:.4f}   {:.4f}    {:.4f}".format(
            rho, ph, phi, q, bw, (1 - ph) ** 2))
    print("moment estimate of rho and the majority it predicts")
    print("  true rho  rho-hat  predicted maj  observed maj")
    for rho in (0.0, 0.25):
        ph, rh, maj = res[rho]
        pred = bb_major(5, ph, max(rh, 0.0))
        print("  {:<8}  {:7.4f}  {:13.4f}  {:12.4f}".format(
            rho, rh, pred, maj))

Single source and citation

Fan, Y. (2026). Condorcet's Jury Theorem with Correlated Errors: When Multi-Model Voting Gets More Accurate and When Shared Origins Make Everyone Wrong Together (Version 1.0.0), Chapter 3: Measuring Correlation: Pairwise Correlation, the Q Statistic, Effective Votes and Colliding Wrong Answers. 5D Library. DOI pending

DOI pending · monograph page goes live after publication · 5DL-B-309 · v1.0.0 · CC-BY-NC-ND-4.0

BibTeX
@book{fan2026309,
  author    = {Fan, Yafu},
  title     = {Condorcet's Jury Theorem with Correlated Errors: When Multi-Model Voting Gets More Accurate and When Shared Origins Make Everyone Wrong Together},
  year      = {2026},
  publisher = {5D Library},
  edition   = {1.0.0},
  note      = {DOI pending}
}

Other chapter answers in this book