• Rebuilt: the 1994 meta-analysis of 13 BCG vaccine trials against tuberculosis, and the 2010 software paper that made it a textbook example, in Python, in R and in Lean.
  • Held: all 164 printed values, and the headline: the vaccine roughly halves the risk (relative risk 0.49) and works better farther from the equator.
  • Did not hold: a few of the 2010 values are not the exact answer; they match only where that software stopped searching. The 1994 claim about study quality cannot be checked, because the quality scores are not in the public dataset.

In 1994 Colditz and colleagues combined thirteen trials of the BCG vaccine against tuberculosis, 357,347 people in all, and reported that it roughly halves the risk (relative risk 0.49, 95% interval 0.34 to 0.70). The trials disagree sharply, and the disagreement follows distance from the equator: the vaccine works well far north and hardly at all in the tropics. The dataset became the textbook example for meta-analysis software.

BcgLatitude reproduces all 164 values printed in the two source papers, three times over, and then asks what a health programme actually needs to know: whether to vaccinate where it is.

How it is built

Python written from scratch, R with the standard metafor package, and Lean 4 with no statistics library at all, run in order by run_all.sh. The run fails on any unexplained mismatch.

nix flake · Python + R + Lean 4 · flake.lock
  1. Pull dat.bcg from a pinned commit of metadat, SHA-256 checked python/01
  2. Fit six tau² estimators and meta-regression, in Python, R (metafor) and Lean python/02 · python/03
  3. Compare three languages against each other and 164 published values python/04
  4. Decide the latitude where the recommendation flips, across twelve conventions python/05 · python/06
  5. Prove the flip between 18° and 19°, kernel-checked (lean/Verdict.lean) python/08

Reproducible. The data file is downloaded from a pinned commit and refused if its SHA-256 differs. Python, R and Lean are pinned in flake.nix and flake.lock, and each run records its versions. Python does not borrow R’s loader: a 130-line reader, python/rda.py, decodes R’s binary format, so a bug in one path cannot hide in both.

Tested. 15 pytest and 4 testthat tests: hand-computed cases, closed forms, library cross-checks, a byte-built file for the data reader, and a check that a deliberately false verdict is rejected by Lean.

The last decimal

All 164 printed values match in Python; 156 in R. The other eight did not match at the true optimum. They matched only where the 2010 software stopped searching: metafor stops its iterative fit once the estimate moves by less than 10⁻⁵, and with year left uncentred, that early stop shifts a printed number in the fourth decimal. So python/utils_meta.py solves each iterative quantity two ways: one ports metafor’s stopping rules, R’s root-finder included, translated from its C source; the other goes to machine precision. The first reproduces the paper, the second gives the true answer, and tightening R’s tolerances makes R land on the second. One value needed the search interval of a 2010 version of the package, because today’s searches differently.

Lean, solving to full precision with gamma functions, t-distributions and an optimiser built from scratch in lean/Meta.lean, agrees with the true optimum rather than the early stops, which confirms the story from a third side. Of 686 quantities Python and R both compute, 677 agree to 10 or more decimals, and the nine others are explained: a nearly singular matrix in one model, and two libraries stopping early by different rules.

Which choice decides the answer

Choice Pooled RR 95% interval Matches 1994?
Ignore the disagreement (fixed effect) 0.65 0.60 to 0.70 no
Five of six tau² estimators 0.49 0.34 to 0.70 yes
Maximum likelihood 0.49 0.35 to 0.69 no
Any estimator with Knapp-Hartung 0.49 0.33 to 0.73 no

The headline barely moves; the interval moves more. And since more than one estimator rounds to the published interval, the paper cannot tell us which one its authors used. The comparison assumes DerSimonian-Laird, the standard of the time, and the README says so.

The decision

The rule: recommend BCG at latitude L if the 95% interval for the predicted relative risk lies entirely below 1. A meta-regression on latitude explains about three quarters of the trials’ disagreement, and the answer flips at 18.2°.

Latitude Predicted RR 95% interval Recommend?
10° (outside the data) 0.96 0.67 to 1.39 no
20° 0.72 0.55 to 0.93 yes
30° 0.54 0.44 to 0.66 yes
40° 0.40 0.32 to 0.51 yes

Across all twelve analysis conventions the flip stays between 17° and 21°. Under a stricter rule, the prediction interval a single new programme should expect, it moves anywhere from 23° to 48°, roughly the Tropic of Cancer to Paris, depending on the estimator. The estimator matters only after you choose the rule, and the rule is the real decision.

Lean’s kernel certifies the flip in exact fractions. From lean/Verdict.lean:

/-- At 19 degrees of absolute latitude the rule recommends BCG. -/
theorem recommend_at_19 : recommend pred_at se_at := by decide +kernel

/-- At 18 degrees it does not: the verdict flips between the two. -/
theorem no_recommendation_at_18 : ¬ recommend pred_below se_below := by decide +kernel

The logarithms and the regression fit enter as declared inputs, and the certificate covers the decision arithmetic only. Latitude is a stand-in, not a cause: it tracks exposure to environmental bacteria, and also when the trials ran (correlation −0.66 with year). A real programme should refit on a measure of exposure.