In 2006 three political scientists mailed postcards to Michigan voters before a primary. One card, “Neighbors,” showed each household whether its members and their neighbours had voted in past elections, and promised an updated list after the vote. Turnout among people who got it was 37.8%, against 29.7% among people who got nothing. The study, Gerber, Green and Larimer in the American Political Science Review (2008), is one of the best-known field experiments in social science. Its data and the authors’ Stata log are public.
NeighborsMailer rebuilds the paper’s two main tables from the raw file, in Python, and checks all 130 numbers against that log, to the last printed digit. Then it asks the question the paper leaves open: send the mailer or not?
How it is built
Five numbered Python scripts, run in order by run_all.sh. The statistics are written out by hand in numpy rather than called from a package, so every convention is visible.
- Pull the authors' Stata file and log from Yale Dataverse, SHA-256 checked python/01
- Build 344,084 voters, 180,002 households, stopped if the design does not match python/02
- Fit Tables 2 and 3, clustered standard errors written out in numpy python/03
- Compare all 130 numbers against the Stata log, rounded as Stata prints python/04
- Decide send or not, three ways, two figures python/05
Reproducible. Python and its packages are pinned in flake.nix and flake.lock, and each run writes output/session_info_python.txt. The download is the authors’ original Stata file, not the archive’s converted copy, and it is refused if its SHA-256 fingerprint is wrong. The build step then checks the data against the published design (344,084 voters, 180,002 households, 10,000 clusters, five arms) and stops if anything is off.
Tested. 7 pytest tests, none needing the data: the hand-written clustered standard error against statsmodels, the degrees-of-freedom factor, the fixed-effects shortcut against a brute-force dummy regression, the household relabelling, the rounding rule, and the arithmetic of the hand-typed reference numbers. The comparison itself is a test: one number off and the run fails.
Three hidden details, and the way past each
The easy part of a replication is the coefficients. Matching the standard errors to seven decimals means knowing which conventions the original software used, and three of them appear in no paper or manual chapter.
The software counts parameters you never see. Two models control for each voter’s neighbourhood, one of 10,000 clusters. Stata’s areg absorbs them, so the 9,999 neighbourhood terms never appear in the output, but it still counts them when it corrects the standard errors for model size. Leave them out and the standard errors move in the fifth decimal, which a seven-decimal log catches. From python/03_fit.py:
G = scores.shape[0]
k = p + k_absorbed
factor = G / (G - 1) * (n - 1) / (n - k)
cov = factor * XtX_inv @ meat @ XtX_inv
The reported R² is not the one you would compute. After absorbing the neighbourhoods, the natural R² on the de-meaned data is 0.0038. Stata reports the full model’s, neighbourhood terms included: 0.1054. The code computes Stata’s on purpose.
Two households have half-numbered IDs. The household ID is stored as a decimal, and two of its 180,002 values are 41511.5 and 43601.5. Casting the column to integers, the obvious cleaning step, merges those two one-person households into their neighbours, the count drops to 180,000, and the clustered standard errors change. From python/02_build_dataset.py:
# hh_id is stored as a float and two of its 180,002 values are not whole
# numbers, so it is relabelled 0..180,001 rather than cast to int, which
# would merge those two households into their neighbours.
df["hh_id_raw"] = df["hh_id"]
df["hh_id"] = pd.factorize(df["hh_id"])[0]
From an effect to a decision
A campaign does not need an effect size. It needs a yes or no. So the project writes the rule down: send the Neighbors card if the lower end of its 95% interval clears 2 percentage points, a stand-in for the campaign’s own cost line, kept as one constant so any other line can be tried.
| Question | Answer |
|---|---|
| Does the method matter? | No. Five ways of computing the interval put the lower bound between 7.47 and 7.62 points, five points clear of the line |
| Who does the rule fail for? | Both ends of voting history: people who voted in 1 of 6 past elections (effect 2.4 points) and in all 6 (4.4 points, but 64% vote anyway) |
| How much of the experiment did it need? | 5% of the Neighbors and control households (6,000) settles the verdict in 98% of 200 redraws |
The subgroup cuts are fifteen comparisons with no multiplicity adjustment, and the README says so: they suggest where to look, not proof. The study was sized to tell four postcards apart and to study subgroups, so the last row is not a criticism of it. It is the number a team planning its next experiment would want.