How a Test Decides ยท five essays

At a Glance

A test starts from a guess it can count, and asks how rare the evidence would be if the guess were true. What it can answer is narrower than what we ask.

  1. 1Significant but WrongWhat does a tiny number assume before it convicts?
  2. 2What It Would Take, In CupsWhat would it take to believe a claim?
  3. 3One Napkin, Two FiresHow many voices are enough to overturn a story?
  4. 4The Seam Does Not Show AnymoreWhere do Fisher's test and Neyman and Pearson's meet?
  5. 5A Weed Called CrazyWhat can a test answer?
Idea12345
The starting guess
Shared, not separate
The line comes first
Not proven is not disproven
Narrower than the question

The starting guess

Every test begins from a default it can count: nothing is going on.

  1. Part 1Elffers had to assume Lucia was like any other nurse: innocent until proven guilty.
  2. Part 2The starting assumption is that she is guessing: the one story that can be counted.
  3. Part 3Miles's story stands until evidence overturns it; the burden fell on Andi.
  4. Part 4Both camps start from the same guess: the driver is keeping pace with traffic.
  5. Part 5Both models began from one guess: the three treatments stop bacteria equally well.

Shared, not separate

Chances multiply only when nothing is shared.

  1. Part 1A ward is a system, not a deck: its shifts share patients, seasons and handovers.
  2. Part 2The cups are shuffled, so a cup's place is independent of how it was poured.
  3. Part 3Four friends paid by one man were one decision spoken four times.
  4. Part 5Three wells on one plate share one pour, so the plates became blocks.

The line comes first

The threshold is set before the evidence comes in.

  1. Part 1A threshold guards the innocent only when the evidence was gathered blind.
  2. Part 2Fisher's usual line was 5 percent; eight right, at 1 in 70, clears it.
  3. Part 3Before any friend speaks, the court decides how many would be enough.
  4. Part 4Neyman and Pearson fix the cutoff beforehand; every driver over it gets the same ticket.
  5. Part 5Move the line from 1 in 100 to 1 in 20, and the 1.8 millimeter gap would count.

Not proven is not disproven

A test that fails to reject has not shown the guess is true.

  1. Part 2Six of eight would have left the test silent, and a silent test proves nothing.
  2. Part 3Failing to reject only means the evidence was not strong enough yet.
  3. Part 4The gun can show a driver was not keeping pace, and never prove one was.
  4. Part 5Not proven different is not proven alike, the way not guilty is not innocent.

Narrower than the question

What a test answers is smaller than what people ask of it.

  1. Part 11 in 342 million was the chance of her shifts if the ward were a deck, not of her innocence.
  2. Part 2A p-value is not the chance that the guessing story is true.
  3. Part 4"Significantly" says a line was crossed, with no size attached.
  4. Part 5A ring in a dish is not a healed wound; only a study of people can take that.