One label on one metric ran the monitoring out of memory

transcript

A developer adds one label. To one metric. And the monitoring system starts running out of memory.

A metric is a name plus labels. Say, a request counter. Each label combination is its own series. Checkout, 200. Checkout, 500. Now someone adds a user ID label. The service has 10 million users. Every user becomes a new series. Up to 10 million. For one metric. And each series keeps a head chunk in memory. Memory goes first.

The whole platform budgets about 50 million series. For everything. This one label can add 10 million. And queries on that metric crawl. Their index sets are huge.

So here's the catch. You can't review every deploy. The defence has to sit at the door.

A user ID on a counter means a series per user. So the ingest gateway caps series. Per tenant, and per metric. Past the cap? Reject it, or drop the label. And alert the team. Need per-user detail? That belongs in logs or traces.

Now one bad deploy can't take the platform down for everyone. Labels must have bounded values. Everything else goes to logs.

Full breakdown in the lecture.