Significant but Wrong

A Dutch nurse took a bad feeling to the chief pediatrician. A six-month-old baby had died overnight at the brick children’s hospital in The Hague, in September 2001 (Figure 1). The death was ruled natural. The nurse worried about who had been on duty: Lucia de Berk, again.

Figure 1. Two hospitals on Sportlaan in The Hague: the Juliana Children’s Hospital (left), its home from 1999 to 2015, where the bad feeling started, and the Red Cross Hospital next door (right), where Lucia had worked two wards. Photos: Joris (2011), CC BY-SA 3.0; 1Veertje (2020), CC BY-SA 4.0; via Wikimedia Commons.

Someone at the hospital ran her shifts through an Excel spreadsheet and called the pattern no coincidence. Each shift became a yes-or-no question: an incident or none. An incident meant a death or a resuscitation. The tally worked like the solitaire count in an earlier essay, where the mathematician Stanislaw Ulam dealt each game fresh from one deck. The hospital’s count was pointed at a person. The same day, the hospital changed the baby’s death to unnatural. Within ten days, two hospitals had linked more than thirty incidents to her. On September 14 the hospital reported five deaths and five other incidents to the police. She was arrested that December and pleaded innocent.

The court used Henk Elffers, a law psychologist, as its statistician. He worked out the chance that so many incidents would fall on her shifts by luck: 1 in 342 million. That is like writing the name of everyone in the United States on a slip, stirring the slips in one drum, and drawing hers first. There was no medical proof of murder. In 2003 a court in The Hague gave her a life sentence for four murders and two attempts, and the statistics were central to it.

The number was significant: far below any test’s threshold, a cut-off set in advance, like the legal limit on a breath test. Elffers told the court it was “not chance.” The number was also wrong. Metta de Noo, a doctor, grew uneasy about the case, and her brother Ton Derksen, a philosopher, wrote a book about it. In 2010, Richard Gill of Leiden University and two fellow statisticians reworked the arithmetic. Their model gave each nurse her own rate of incidents, and on the same counts the model gives about 1 in 49.

As a biostatistician, I compute numbers like that one, and a small number tells me nothing until I know what the count assumed.

Say you are a workaholic who decides to practice il dolce far niente, the Italian sweetness of doing nothing. Merriam-Webster calls it “pleasant relaxation in carefree idleness” (Figure 2). Yours is twenty idle minutes after lunch, no phone, no list. On a sticky note by the sink you tally each weekday, and after four weeks it reads 3 rests out of 20. A mind that will not sit still treats each missed afternoon as a fresh deal, and reaches a verdict: bad at rest. But the afternoons were never fresh deals. The misses bunched into the same two weeks, ten of the 17, when a family member had an emergency and a deadline came due at work. On Sunday evening, you peel off the note with a soft rip and stop trying. The people at home get the restless version of you back.

Figure 2. Dolce Far Niente, by John William Waterhouse (1880). Public domain, via Wikimedia Commons.

What is the sticky note’s 3 rests out of 20 assuming, before it ends the afternoon rest?

What the Count Assumed

Elffers drew the children’s ward as a table of four boxes in 2002. Over the months he studied, the ward ran 1,029 shifts, and Lucia worked 142 of them. The ward logged 8 incidents, and all 8 fell on her shifts.

Before that table could say anything, Elffers had to assume something: that Lucia was like any other nurse, and incidents land on shifts by chance. A jury starts the same way, innocent until proven guilty. Statisticians give the starting guess a name.

Elffers checked his guess with Fisher’s exact test, the test built for a woman tasting eight cups of tea. The test asks how often chance alone would put so many incidents on one nurse’s shifts. On the children’s ward, shuffle 8 red cards into a deck of 1,029 shifts and deal them at random. Count the ways all 8 can land on Lucia’s 142, and divide by the ways they can land anywhere. The answer is 1 in 9.0 million. Any of the ward’s 27 nurses might have drawn such a run, so Elffers multiplied that chance by 27. The product was 1 in 335,000. No one gasps when somebody in a 27-person office wins its raffle.

Elffers had three tables, not one. Lucia had also worked two wards of the Red Cross Hospital, numbered 41 and 42.

On ward 41, Lucia worked 1 shift of 366. One of the ward’s 5 incidents fell on that shift. Any one shift holds an incident 5 times in 366, or 1 in 73. On ward 42, she worked 58 of 339 shifts. Chance alone would put about 2.4 of its 14 incidents on hers. She had 5, and five or more comes up 1 time in 14. Elffers then multiplied the children’s ward’s 1 in 335,000 by ward 41’s 1 in 73 (Figure 3, first tab). Times ward 42’s 1 in 14, the product reaches 1 in 342 million, her name drawn first from the drum. I reran each chance from his table, and my totals match.

That multiplication rests on four assumptions (Table 1).

Table 1. The four assumptions behind Elffers’s 1 in 342 million, in plain words. Made for this piece.

Figure 3. One ward table, counted three ways. Tap each count to see its formula. Built for this piece.

Does a ward work like a deck of cards, as all the assumptions but the blind count require?

What a Ward Shares

Ulam reshuffled his fifty-two red and black cards before every game, so one game could not affect the next. A deck is not a system.

The biologist Ludwig von Bertalanffy defined a system as interacting elements, and said what its opposite looks like: when “the elements behave independently,” there is no system. The systems thinker Russell Ackoff called a system a whole “that cannot be divided into independent parts.” Multiplying the wards’ chances, and treating each shift as a separate deal, works only if the ward splits into independent parts, which a system cannot do.

The environmental scientist Donella Meadows gave three things to look for in a system: its parts, their interconnections and what the whole is for. A ward has all three, and its parts and interconnections break three of Elffers’s assumptions.

• Fresh shifts. A ward’s interconnections run through its shifts, which share patients, seasons and handovers, the way wells on one lab plate share one pour. A sick baby’s night of crying carries into the morning shift. On any ward, Gill wrote, events come “strongly clustered.” One hard week can put several incidents on the same few shifts, and on whoever works them. A coworker’s three missed deadlines in one flu week are one cause, not three failures.
• Separate wards. The ties also run between the wards. One search, started by one bad feeling, built all three tables around one nurse, so they are not independent. In 2010, Gill and his co-authors Piet Groeneboom and Peter de Jong pointed to the “absurd consequence” of multiplying the three chances. By that arithmetic, a nurse grows guiltier each time she changes hospitals. Add the three wards into one table instead, 1,734 shifts with 201 of them Lucia’s. Chance alone puts about 3 of the 27 incidents on her shifts. She had 14. Fourteen or more comes up 1 time in 3.8 million (Figure 3, second tab), still tiny, but one count, not three multiplied.
• Same rate. The parts differ. Nurses with different training “do get different shifts,” Gill wrote, so one nurse’s count can differ from another’s with no one to blame. Gill’s team gave each nurse her own rate, drawn from an assumed spread in which some nurses run at twice the average or more. I reran their model from its formula. Take a nurse who works Lucia’s 201 shifts, at a rate no one knows. She reaches 14 or more incidents 1 time in 49 (third tab), about one week in a year: unusual, not proof.

Meadows’s third thing is what the whole is for. Ackoff called it the essential property: what the whole can do that no part can do alone. A ward for very sick children exists to keep them alive, and no single nurse can do that. Some children die anyway, and each death still falls on somebody’s shift. A count that never asks what the ward is for reads every death as a clue. A fair count starts from the ward, its shared causes and its own rate of deaths, before it looks at any one nurse.

One assumption is left, the blind count: every incident was dealt before anyone went looking for it.

How the Count Was Dealt

In the case file, an incident was whatever the hospital called one. Gill and his co-authors found incidents “relabeled without clear reasons,” and a data collection that “rested for a large part on memory.” Staff searched the records after the bad feeling, knowing whom to look for. “Incidents outside shifts of Lucia were discarded.”

A death no one could explain became suspicious if Lucia had been present. In two cases, by Gill’s account, only one expert called the death unusual, and only because Lucia had been there. Each new suspicious case made her presence look more meaningful, sending the search back through more records. The search fed itself, around a loop (Figure 4).

Figure 4. The search loop. Steps as Gill, Groeneboom and de Jong describe them; the loop is this piece’s reading. Drawn for this piece.

Ulam dealt his games before he counted them. The hospital counted cards it had already turned face up and sorted. That count was like someone at a kitchen table who looks through the deck, keeps the hearts, and then asks how rare it is to hold so many hearts. The hand was picked, not dealt.

The cluster also came before the question. Elffers’s multiplier of 27, one for each nurse on the ward, is fair when the question comes first. Here the bad feeling had already picked the nurse. Gill’s team named the trap: “testing a hypothesis using the same data which suggested that hypothesis.” Across a country’s wards, some nurse will have a run of bad shifts by chance alone. In a big group chat, two friends often share a birthday no one planned. A fair count asks how often some nurse, on some ward, would look this unlucky.

Even the edge of the count was dealt after the fact. Elffers’s tables took in Lucia’s wards, during her months. The hospital director had renamed the children’s ward, and he never told the investigators that two years before, under its old name, it had logged even more incidents. To him, the old name likely belonged to the past. Two years wider, the count would hold a bigger cluster than hers.

A test is meant to guard against all these traps with a threshold, set before the evidence comes in and low enough to rarely convict the innocent. Convicting the innocent has a name, a Type I error. The threshold guards against it only when the evidence was gathered blind. Evidence gathered inside the loop had already passed through the suspicion under test, so the loop, not chance, carried it over the threshold.

Even a fair threshold gives only a pass or a fail. What a pass means, Elffers had to say in court.

What a Significant Count Cannot Say

Elffers told the court: “Your honour, it was not chance. The rest is up to you.” He meant the cluster was unlikely to be bad luck, not that it proved cause. Lucia might work harder shifts, or someone might have made her look guilty. Elffers’s words were easy to mishear as a 1 in 342 million chance that she was innocent.

A chance of the evidence and a chance of innocence are different numbers. A test gives the chance of evidence this extreme if the starting guess is true. The 1 in 342 million was the chance of her shifts if the ward were a deck. Her innocence turned on what that count left out: shared causes, nurses who differ, a ward where very sick children die, and a search that picked her first. The yellow sticky note by the sink holds the same trap. Its 3 rests out of 20 say nothing about a person until the two hard weeks are marked on it. Read alone, they end the habit.

In 2004 the appeals court went further: seven murders and three attempts. It said its verdict did not rest on statistics. Gill’s team showed the verdict still leaned on incidents judged “very improbable.” Only one or two murders had to be proven on their own.

On April 14, 2010, a court acquitted Lucia de Berk of every charge. The deaths were found natural. She had spent six years in prison. She died in 2025, at 63 (Figure 5). The hand had been dealt by a ward, not a deck, and the spreadsheet had a column for only one nurse.

Figure 5. Lucia de Berk, reading her mail. Photo: Carole Edrich, CC BY-SA 3.0, via Wikimedia Commons.

A Closing Invitation. The bad feeling stood for a count that picked its cards before it counted them. A fair count treats a ward, a week or a habit as a system, not a deck, and sets its threshold first.

1. Flip the odds. Now, recall the last true-crime show or news trial whose “1 in a million” settled an argument with your partner or a friend. Say those odds out loud two ways: the chance of the evidence if the suspect were innocent, then the chance of innocence. Which did you hear first?
2. Count the other columns. This week, think of the coworker or classmate whose name comes up whenever something goes wrong, like the last missed deadline or the shared fridge that began to smell. How many others were there each time, and did anyone write their names down too? Was the pattern counted before or after someone was worried?
3. Find the first miss. On Sunday evening, before you score the week, scroll back through your phone’s calendar and find the first afternoon you ate lunch at your desk or took a work call instead of taking the break. Did one missed break make the next easier to skip? Which break next week will you guard first, and who will you tell?

A bad feeling is worth saying out loud. Before it is counted, every card goes back in the deck.

Worth remembering Save as image

Odds multiply only when no card depends on the one before. Days, shifts and people rarely work that way: they share a boss, a season, a sleepless night. Before bad luck convicts anyone, yourself included, ask what the cards shared.