510,000 vs 490,000 looks like noise. It's 20 standard deviations.

transcript

Your experiment shows 510,000 users against 490,000. Looks like noise. It's 20 standard deviations. It's a bug.

The candidate's guard on results. Start with a million users, split fifty fifty. You expect 500,000 in each arm. Chance moves that by about 500. The square root of a million, times a half, times a half.

One standard deviation here is 500 users. Treatment is 10,000 over its expected 500,000. 10,000 over 500. That's 20 standard deviations. Not chance. A bug.

Usually, exposures are logged on one variation's code path. And not the other. Or one variation crashes before it logs. Either way, the result can't be trusted. Neither can its winner.

So the candidate adds an automatic check. The counts still show. But no winner. Results hidden: sample ratio mismatch. With the likely cause, so their engineer knows where to look.

People make launch decisions from one screenshot. A confident wrong answer is the worst thing you can show them. Check the split first.

Full breakdown in the lecture.