One ad click, four chances to bill it twice

transcript

One ad click. Four places it can turn into two. And advertisers pay per click.

The client. A double tap, or a browser retrying the redirect. The tracker. It writes to Kafka, the ack times out, it writes again. The processor crashes after counting. Before saving how far it read. And the sink. A retried increment adds the count twice.

So which feature fixes all four? None of them. Exactly once is a chain. Every link must be safe to repeat.

Link one. A click ID, signed into the link when the ad is served. Two. Kafka's idempotent producer. Tracker retries don't duplicate. Three. Flink drops any click ID it saw in the last hour. Four. Counts, seen IDs and read position, saved in one checkpoint. Five. The sink overwrites. It never adds.

An increment says plus 5. Retry it, and it counts twice. Overwrite says the count is 412. Write it twice? Still 412.

And behind it all, an hourly batch recomputes from raw clicks. So even a streaming bug can't reach an invoice.

Full breakdown in the lecture.