• Rebuilt: 3,823 Upworthy headline tests from 2013 to 2015, from the public archive, each decided again by the expected-loss rule most Bayesian A/B tools use.
  • Held: every count matches the archive’s data paper, and the rule picks the editors’ version in 72.5% of the 896 tests with a recorded winner.
  • Did not hold: agreement in every test. In most of the other quarter the editors had also passed over the dashboard’s leader, so the split looks editorial, and it costs about 0.04 click-rate points a test on average.

Between 2013 and 2015 the news site Upworthy ran thousands of A/B tests on its headlines. Each showed versions of a headline to random visitors and counted clicks, and an editor watched a dashboard and decided when to stop and which version to run. The archive of those tests is public.

HeadlineExpectedLoss replays every test under the rule most Bayesian A/B tools now use, expected loss, and asks three questions a product team asks: which version, when to stop, and how much data the decision needed. It is a study of decisions, not of headlines.

The rule

A significance test asks whether a difference is real. The expected-loss rule asks: if I run this version now and I am wrong, how many clicks do I give up, on average? Each version’s click-rate gets a Beta posterior; the expected loss of a version is its average shortfall against whichever is truly best. Run the version with the smallest loss, and stop once that loss falls below a threshold of caring, ε, here 0.1 click-rate points.

One test, worked through: version A had 43 clicks on 9,849 impressions, version B 40 on 9,738. A is better with probability 0.61, its expected loss is 0.026 points, B’s is 0.052. Both are under ε, so the rule runs A and stops. The editors ran B, a disagreement that sounds bad on a scoreboard and costs less than the threshold once the uncertainty is priced in.

How it is built

Seven numbered Python scripts and a Lean proof, run in order by run_all.sh.

nix flake · Python + Lean 4 · flake.lock
  1. Pull the Upworthy Research Archive, checked against its published SHA-256 python/01
  2. Build one row per headline version, the cache window flagged and dropped python/02
  3. Posterior Beta posteriors and expected loss, two priors, 20,000 draws per test python/03
  4. Compare 511 checks against the data paper and exact formulas python/04
  5. Decide which version, when to stop, how much data python/05
  6. Prove all 99 two-version verdicts in exact fractions (lean/Decision.lean) python/06 · python/07

Reproducible. Python and Lean are pinned in flake.nix and flake.lock, the simulation uses a fixed seed, and each run records its versions. The download is refused if its SHA-256 differs from the one the archive publishes. In 2024 the archive warned that a web cache likely broke randomization for a stretch of tests; the older file does not mark them, so the build rebuilds the flag from the archive’s dates and drops those 1,050 tests.

Tested. 10 pytest tests, none needing the data: the formulas against numerical integration, the simulation against the formulas, the traffic-thinning step, the featured test’s verdict, and a check that a wrong count really stops the run. The comparison step checks 511 quantities, counts against the archive’s data paper and simulation against exact formulas, with 0 mismatches.

Two priors. A flat prior lets the data speak. A shrunk prior, learned once from the whole archive by empirical Bayes, pulls each version halfway toward its test’s average, which tames the lucky early leaders a dashboard over-rewards. They agree with the editors 72.5% and 72.9% of the time, so the conclusions are not an artefact of the prior.

Three answers

Question Answer
Which version? The rule picks the editors’ version in 72.5% of 896 tests. The editors themselves ran the dashboard’s leader only 73.4% of the time, so most disagreements look editorial, and they cost about 0.04 points on a typical 1.3% click-rate
When to stop? At the traffic each test got, the rule was ready to stop in 74% (flat) to 82% (shrunk) of tests
How much data? With half the traffic, the rule stops about half the tests, and its early pick matches its final pick 85% of the time

ε is a business choice, so every result is shown across five thresholds, from 0.02 points, which stops about one test in four, to 0.2, which stops nearly all.

Every two-version verdict, proved

For two versions and a flat prior, the probability and the expected loss are exact sums of fractions. Lean computes all 99 two-version verdicts that way and proves they add up to the reported counts: 36 of 47 agree with the editors, 89 of 99 pick the dashboard’s leader. From lean/Decision.lean:

theorem verdicts_add_up : summarize tests = expected := by decide +kernel

theorem featured_pick_expected_loss_under_epsilon :
    Frac.lt (verdict featured).lossA epsilon = true := by decide +kernel

Change any count by one and the proof fails. No outside maths library is used, so what has to be trusted is Lean’s core and the two formulas as written. Tests with three or more versions have no closed form and stay with the simulation, which sits within four standard errors of Lean’s exact values on all 99.

The archive records what editors chose, not why, and not what a headline did once live, so nothing here prices a click. Only the archive’s exploratory sample is used: its curators ask that the confirmatory sample be touched only with a pre-registered plan.