5D Library › Two-Scale Renormalization, Occupancy Entropy and Thrust-Wave Superposition in Discrete Cyclical Networks · 中文

Chapter 6 Cleaning Out Counterexamples and Spurious Correlations: The Barnum Effect, Multiple Testing and Data Snooping

The first five chapters built mathematics: definitions, theorems and simulations. As soon as someone links these structures to calendar labels and says “opposition months are more volatile” or “sector rotation is fiercer in such-and-such a year”, the question becomes empirical, and empirical questions have their own traps. This chapter discusses three: vague descriptions that feel “accurate” (the Barnum effect); so many patterns tested that some are bound to come out significant by chance (multiple testing); and repeated trials on the same data followed by a report of the best (data snooping). 5D Library volume 33 has treated falsifiability and p-hacking systematically; here we discuss only what bears directly on calendar labels.

“So accurate”: the Barnum effect

Evidence

Forer’s 1949 classroom demonstration. Forer had students take a personality test and then gave each of them a “personal analysis” said to be based on the results, asking them to rate how well it described them on a scale from 0 to 5. In fact everyone received the same description, made up of general statements such as “you have a great deal of unused capacity”. The students’ mean rating was about 4.3 (5 being the most accurate; secondary sources give 4.26 or 4.3) (Forer 1949). Dickson and Kelly reviewed the many later studies in 1985: this “fallacy of personal validation”, later generally called the Barnum effect, recurs across populations, and the more favourable a description is, and the more it is presented as “written specially for you”, the more readily people accept it (Dickson and Kelly 1985).

What the Barnum effect means for this book is that a person’s feeling that a calendar-based description is “accurate” gives almost no information about whether the description is valid, because equally vague descriptions are “accurate” for almost everyone. To test a description, people must pick out their own from several descriptions without knowing which is theirs: a double-blind matching test.

Double-blind tests of astrological prediction

Evidence

Carlson’s 1985 test in Nature. Carlson worked with an American astrological organisation to agree in advance on a protocol the astrologers considered fair. Twenty-eight astrologers took part. For each subject’s natal chart, an astrologer chose which of three California Psychological Inventory (CPI) profiles belonged to that subject; one profile was genuine and the other two were taken at random from other subjects. Of 116 usable judgements, the first choice was correct about 34 % of the time, no different from the one-third expected by chance and far below the at least one-half the astrologers had predicted; nor were the astrologers more accurate on judgements they rated as confident (Carlson 1985). Another part of the study asked subjects to pick their own chart interpretation out of three; the author considered that part inconclusive, because subjects could not even pick out their own CPI profiles.

Evidence

Dispute and further tests. Ertel reanalysed Carlson’s data in 2009, arguing that Carlson’s design and statistics were flawed and that with a different analysis the astrologers’ three-way matching was slightly better than chance, at marginal significance, and Carlson’s 10-point rating test gave a significant positive result; he also said this was not enough to confirm astrology, only that Carlson’s negative verdict could not stand (Ertel 2009). Dean and Kelly reviewed further tests in 2003: a large study of “time twins”, people born less than five minutes apart, found none of the similarities their charts predicted, and a meta-analysis of more than forty controlled studies found that astrologers did no better than chance, even on a task as basic as judging extraversion (Dean and Kelly 2003).

Two points must be stated. First, these tests all concern Western natal-chart astrology, not stem-and-branch fate calculation. The author has found no comparable double-blind test of stem-and-branch predictions, so one cannot say that stem-and-branch fate calculation “has been refuted by double-blind tests”; one can say only that astrological prediction of the same kind has shown no better-than-chance ability in double-blind tests, while stem-and-branch fate calculation itself has not been tested in this way. Second, this book tests no prediction of fate; Chapter 8 tests one specific statistical hypothesis about calendar labels and monthly market volatility, which has nothing to do with divinatory inference.

One hundred and seventy-six tests: “regularities” in pure noise

Calendar labels are especially good at producing spurious regularities, because each month carries many labels at once: year stem, year branch, month stem, month branch, the positions of the year and of the month in the sixty-term cycle, and the difference between month branch and year branch (d=6d = 6 is “opposition”). Every value of every label can be compared with the other months.

Calculation

The program generates 2,000 “histories”, each of 96 years × 12 months = 1,152 independent standard normal values, with no calendar effect whatever. In each history every value of 7 labels (176 “patterns” in all) is tested once: the mean of months with that value against the mean of the rest. Three things are recorded: how many patterns per history are “significant” at uncorrected p<0.05p < 0.05; the share of histories with at least one “discovery”, uncorrected and with Bonferroni and Benjamini–Hochberg correction; and how the most significant pattern in the first 48 years does in the last 48.

Calculation

# Pattern mining on PURE NOISE: 96 x 12 = 1,152 independent N(0, 1) months. Every value
# of seven calendar labels is tested against all other months (normal-approximation
# t test): 176 tests per history.
import math, random
from statistics import NormalDist
YEARS, Y0 = 96, 6                       # first year = index 6 in the cycle (Geng-Wu)
PHI = NormalDist().cdf

def labels(T):
    y, m = divmod(T, 12)
    yk = (y + Y0) % 60; mk = (12 * (y + Y0) + m + 2) % 60
    return {"year stem": yk % 10, "year branch": yk % 12, "year cycle no.": yk,
            "month stem": mk % 10, "month branch": mk % 12, "month cycle no.": mk,
            "offset d": (mk % 12 - yk % 12) % 12}

LAB = [labels(T) for T in range(12 * YEARS)]

def tests(x, idx):
    """Return list of (name, t, p) for every class of every label over months idx."""
    out = []
    for fam in LAB[0]:
        groups = {}
        for i in idx:
            groups.setdefault(LAB[i][fam], []).append(x[i])
        tot, n = sum(x[i] for i in idx), len(idx)
        ss = sum(x[i] ** 2 for i in idx)
        for c, g in groups.items():
            k = len(g); m1 = sum(g) / k; m0 = (tot - sum(g)) / (n - k)
            var = (ss - tot * tot / n) / (n - 1)
            t = (m1 - m0) / math.sqrt(var * (1 / k + 1 / (n - k)))
            out.append((f"{fam} {c}", t, 2 * (1 - PHI(abs(t)))))
    return out

def bh(ps, q=0.05):
    s = sorted(ps); m = len(ps)
    k = max([i + 1 for i, p in enumerate(s) if p <= q * (i + 1) / m], default=0)
    return k

def experiment(H=2000, seed=1995):
    """H pure-noise histories: discoveries per history and the best in-sample pattern."""
    rng = random.Random(seed); counts, best_in, best_out = [], [], []
    fw_bonf = fw_bh = 0
    for h in range(H):
        x = [rng.gauss(0, 1) for _ in range(12 * YEARS)]
        res = tests(x, range(12 * YEARS)); ps = [p for _, _, p in res]; m = len(ps)
        counts.append(sum(p < 0.05 for p in ps))
        fw_bonf += any(p < 0.05 / m for p in ps); fw_bh += bh(ps) > 0
        half = 12 * YEARS // 2
        second = {nm: t for nm, t, _ in tests(x, range(half, 12 * YEARS))}
        first = [r for r in tests(x, range(half)) if r[0] in second]
        name, t1, _ = max(first, key=lambda r: abs(r[1])); t2 = second[name]
        best_in.append(abs(t1)); best_out.append(t2 * (1 if t1 > 0 else -1))
    return m, counts, fw_bonf, fw_bh, best_in, best_out

if __name__ == "__main__":
    H = 2000
    m, counts, fw_bonf, fw_bh, best_in, best_out = experiment(H)
    raw = sum(counts); fw_raw = sum(c > 0 for c in counts)
    print(f"tests per history: {m}; histories: {H}")
    print(f"mean 'significant' patterns at p < 0.05, uncorrected: {raw / H:.1f}")
    print("share of histories with at least one discovery:")
    print(f"   uncorrected {fw_raw / H:.0%}, Bonferroni {fw_bonf / H:.1%},"
          f" Benjamini-Hochberg {fw_bh / H:.1%}")
    print(f"best pattern in the first 48 years: mean |t| = {sum(best_in) / H:.2f}")
    print(f"same pattern in the last 48 years:  mean t = {sum(best_out) / H:+.2f};"
          f" still t > 1.96 in {sum(t > 1.96 for t in best_out) / H:.1%}")
# tests per history: 176; histories: 2000
# mean 'significant' patterns at p < 0.05, uncorrected: 8.8
# share of histories with at least one discovery:
#    uncorrected 100%, Bonferroni 3.6%, Benjamini-Hochberg 3.7%
# best pattern in the first 48 years: mean |t| = 2.85
# same pattern in the last 48 years:  mean t = +0.00; still t > 1.96 in 2.5%

On average each history has 8.8 patterns “significant” without correction, matching 176×0.05=8.8176 \times 0.05 = 8.8, and every history has at least one such “discovery”. Bonferroni correction lowers the significance level of each test to 0.05/1760.05/176, keeping the probability of at least one false discovery at no more than 5 % (Dunn 1961); the procedure Benjamini and Hochberg proposed in 1995 controls instead the expected proportion of false discoveries among the discoveries (the false discovery rate), and with many tests it usually has more power than Bonferroni (Benjamini and Hochberg 1995). In pure noise, both corrections bring the share of histories with at least one “discovery” below 5 %: 3.6 % with Bonferroni and 3.7 % with Benjamini–Hochberg (each with a Monte Carlo error of about ±0.4 %). When every null hypothesis is true, both procedures control this same share, so they barely differ here; the difference between them shows only when some effects are real.

The last two lines are the most telling. The most significant pattern in the first 48 years has a mean |t||t| of 2.85, which looks convincing; the same pattern in the last 48 years has a mean tt of 0.00, and only 2.5 % remain significant (t>1.96t > 1.96), exactly what chance predicts.

Left: in 2,000 pure-noise histories, the number of patterns per history that are “significant” (p < 0.05) among 176 tests; the dashed line is the mean, 8.8. Right: |t| of the most significant pattern in the first 48 years against its t in the last 48; the gold dashed line marks t = 1.96.

Data snooping and backtest overfitting

Evidence

Data snooping in financial data. White pointed out in 2000 that when the same data are used again and again to select models, the performance of the best model must be compared with the null distribution of “the best among all the models considered”, and he proposed a corresponding “reality check” (White 2000). Sullivan, Timmermann and White applied the method to calendar effects in stock returns in 2001: they considered a large universe of calendar rules and found that many rules look significant on their own, but that once data snooping over the whole universe of rules is taken into account, even the best calendar rule is no longer significant (Sullivan et al. 2001). Bailey and colleagues showed mathematically in 2014 that with enough strategies tried, a very high backtested Sharpe ratio can be obtained on pure noise, and that such strategies perform poorly out of sample (Bailey et al. 2014).

A statement such as “I backtested a hundred years of data with calendar labels and found a regularity” therefore cannot be evaluated unless it says how many labels, outcomes and time windows were tried in all. Step 3 of the Chapter 8 checklist addresses exactly this.

The null distribution must respect time structure

The idea of a permutation test is simple: if labels and outcome are unrelated, then reassigning the labels at random should give statistics “just as likely” as the original, and the position of the original statistic in the permutation distribution is the pp value (Good 2005). But volatility in financial time series comes in clusters: large moves tend to be followed by large moves (Bollerslev 1986). A permutation that shuffles the months completely then destroys this structure, and the null distribution is wrong.

Calculation

The program generates monthly returns from a GARCH(1,1) model with volatility clustering (illustrative parameters) and no calendar effect, with log⁡|r|\log|r| as the outcome. It compares two labels: label A is the “opposition month”, one month per year, scattered across the years; label B is “every month of a zi year”, 12 consecutive months every 12 years. For each label it compares three null distributions: shuffling the months completely; shifting the whole label sequence circularly; and (for label A only) redrawing the flagged month at random within each year (a stratified permutation, using within-year contrasts). Each is repeated 1,000 times, and the share of false rejections of the null at the 5 % level is recorded.

Calculation

# Which null? GARCH(1,1) returns (alpha 0.10, beta 0.88; illustrative), NO calendar
# effect; outcome log|r|. Label A: opposition month (one per year). Label B: all months of
# Zi years (12 in a row). Nulls: shuffle all months; circular shift of the labels; and
# (A only) redraw the flagged month within each year using within-year contrasts.
import math, random
YEARS, Y0 = 96, 6
L = 12 * YEARS

def flags(kind):
    out = []
    for T in range(L):
        y, m = divmod(T, 12); yb = (y + Y0) % 12; mb = (m + 2) % 12
        out.append((mb - yb) % 12 == 6 if kind == "A" else yb == 0)
    return out

def garch(rng, a=0.10, b=0.88):
    s2, r = 1.0, []
    for _ in range(L):
        x = math.sqrt(s2) * rng.gauss(0, 1); r.append(x)
        s2 = (1 - a - b) + a * x * x + b * s2
    return r

def stat(y, idx, tot):
    s = sum(y[i] for i in idx); k = len(idx)
    return s / k - (tot - s) / (L - k)

def p_value(d0, null):
    return (1 + sum(abs(d) >= abs(d0) for d in null)) / (1 + len(null))

if __name__ == "__main__":
    rng = random.Random(144); sims = 1000
    for kind in ("A", "B"):
        f = flags(kind); idx = [i for i in range(L) if f[i]]
        rej = {"shuffle": 0, "shift": 0, "within-year": 0}
        for _ in range(sims):
            y = [math.log(abs(v) + 1e-12) for v in garch(rng)]; tot = sum(y)
            d0 = stat(y, idx, tot)
            null = [stat(y, rng.sample(range(L), len(idx)), tot) for _ in range(199)]
            rej["shuffle"] += p_value(d0, null) <= 0.05
            null = [stat(y, [(i + s) % L for i in idx], tot) for s in range(1, 144)]
            rej["shift"] += p_value(d0, null) <= 0.05
            if kind == "A":
                c = [[12 / 11 * (y[12 * j + m] - sum(y[12 * j:12 * j + 12]) / 12)
                      for m in range(12)] for j in range(YEARS)]
                w0 = sum(c[j][i % 12] for j, i in enumerate(idx)) / YEARS
                null = [sum(c[j][rng.randrange(12)] for j in range(YEARS)) / YEARS
                        for _ in range(199)]
                rej["within-year"] += p_value(w0, null) <= 0.05
        print(f"label {kind}: {len(idx)} flagged months; false-positive rate at 0.05:")
        print("   " + " | ".join(f"{k} {v / sims:.1%}" for k, v in rej.items()
                                 if kind == "A" or k != "within-year"))
# label A: 96 flagged months; false-positive rate at 0.05:
#    shuffle 4.4% | shift 5.9% | within-year 5.7%
# label B: 96 flagged months; false-positive rate at 0.05:
#    shuffle 9.7% | shift 4.9%

For label A all three null distributions give false-positive rates near 5 % (4.4 %, 5.9 % and 5.7 %, each with Monte Carlo error of about ±0.7 %), because opposition months are scattered across the years and volatility clustering hardly affects them. For label B, shuffling the months completely raises the false-positive rate to 9.7 %, almost twice the nominal level: twelve consecutive months share the same stretch of high or low volatility, and once they are shuffled the null distribution is too narrow. Circular shifting keeps the time structure and gives 4.9 %. The protocol of Chapter 8 therefore uses the stratified permutation as its primary test and circular shifting as a sensitivity analysis.

Chapter summary

The Barnum effect shows that “feeling accurate” is not evidence: Forer’s students gave a description identical for all of them a mean of about 4.3 out of 5. In Carlson’s 1985 double-blind test the astrologers’ first choices were right about 34 % of the time, no different from the chance one-third; Ertel’s reanalysis is disputed, and Dean and Kelly’s review and the time-twins study are likewise negative; for stem-and-branch fate calculation the author found no comparable test. In pure noise without any effect, 176 tests yield on average 8.8 “significant” patterns and every history has a “discovery”; the most significant pattern of the first half has a mean tt of 0.00 in the second. Data snooping must be corrected against a null distribution over “all models considered”, and the null distribution must also respect time structure, or else for contiguous labels the false-positive rate can double. The next chapter returns to the documentary sources of these rules.

References

Bailey, David H., Jonathan M. Borwein, Marcos López de Prado, and Qiji Jim Zhu. 2014. “Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance.” Notices of the American Mathematical Society 61 (5): 458–71. https://doi.org/10.1090/noti1105.
Benjamini, Yoav, and Yosef Hochberg. 1995. “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society: Series B (Methodological) 57 (1): 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x.
Bollerslev, Tim. 1986. “Generalized Autoregressive Conditional Heteroskedasticity.” Journal of Econometrics 31 (3): 307–27. https://doi.org/10.1016/0304-4076(86)90063-1.
Carlson, Shawn. 1985. “A Double-Blind Test of Astrology.” Nature 318 (6045): 419–25. https://doi.org/10.1038/318419a0.
Dean, Geoffrey, and Ivan W. Kelly. 2003. “Is Astrology Relevant to Consciousness and Psi?” Journal of Consciousness Studies 10 (6–7): 175–98.
Dickson, D. H., and I. W. Kelly. 1985. “The ‘Barnum Effect’ in Personality Assessment: A Review of the Literature.” Psychological Reports 57 (2): 367–82. https://doi.org/10.2466/pr0.1985.57.2.367.
Dunn, Olive Jean. 1961. “Multiple Comparisons Among Means.” Journal of the American Statistical Association 56 (293): 52–64. https://doi.org/10.1080/01621459.1961.10482090.
Ertel, Suitbert. 2009. “Appraisal of Shawn Carlson’s Renowned Astrology Tests.” Journal of Scientific Exploration 23 (2): 125–37.
Forer, Bertram R. 1949. “The Fallacy of Personal Validation: A Classroom Demonstration of Gullibility.” Journal of Abnormal and Social Psychology 44 (1): 118–23. https://doi.org/10.1037/h0059240.
Good, Phillip I. 2005. Permutation, Parametric, and Bootstrap Tests of Hypotheses. 3rd ed. Springer.
Sullivan, Ryan, Allan Timmermann, and Halbert White. 2001. “Dangers of Data Mining: The Case of Calendar Effects in Stock Returns.” Journal of Econometrics 105 (1): 249–86. https://doi.org/10.1016/S0304-4076(01)00077-X.
White, Halbert. 2000. “A Reality Check for Data Snooping.” Econometrica 68 (5): 1097–126. https://doi.org/10.1111/1468-0262.00152.

Fan, Y. (2026). Two-Scale Renormalization, Occupancy Entropy and Thrust-Wave Superposition in Discrete Cyclical Networks: Algebraic Topology and Monte Carlo Verification on a Twelve-Node Ring Lattice [离散周期网络中的双尺度重整化、占位数分布熵与推力波叠加定理] (Version 1.0.0). 5D Library. https://doi.org/10.5281/zenodo.23796761 · Licensed under Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).