One line on top of the prompt killed the cache

transcript

One line of text quietly killed your cache. And by the evening, it took your rate limit too.

Your assistant sends a 3,000-token system prompt. Every single turn. The provider caches that prefix. Reading it back costs a twentieth. About 90 percent of your input comes from cache. Cheap and fast.

Monday, 10:30. A new prompt adds one tiny line on top. The current time. To the second. 10:35. Cache reads drop to almost zero. Nobody has a graph. 20:30, peak hour. The provider starts answering 429.

Here's the catch. A cache hit needs an exact prefix match. The clock changes every second. So every prefix is brand new.

Each conversation sends 18,000 system-prompt tokens. They were read at 10 cents a million. Now they're rewritten on every call. At 2 dollars 50.

Worse, the rate limit counts cache writes. Not reads. With the cache working: about 290,000 counted tokens a minute. With no reads: about 3 point 4 5 million. Over ten times more.

The fix is boring. Move the time down. Into the newest turn, below the breakpoint. Rounded to what's needed. Then alert on cache-read share, per prompt version.

Stable content on top. Anything that changes goes at the bottom. Your cache is a prefix. Guard it.

Full breakdown in the lecture.