Your Wi-Fi drops mid-reply. The GPU never notices.

transcript

Your Wi-Fi drops halfway through a chatbot reply. And the GPU writing it never even notices. Here's how.

Meera walks out of a café. At token 212, her Wi-Fi drops. Two seconds later, she's on mobile data. Talking to a different server. The reply just pauses. Then carries on, from token 213.

The obvious fix: resend the message. Then the GPU writes the whole reply again. It already paid for those tokens. Instead, number every token. And ask for everything after 212.

Every token the GPU writes is appended to the reply's log. And goes out to the phone as a numbered event. The phone reconnects and says: last event, 212. Any edge can forward that. The log replays 213 onwards, then goes live. The GPU never noticed.

Then she taps Stop, at token 260. Those 260 tokens are saved. Her next turn sees exactly what she saw. And the GPU frees that reply's memory at the very next step.

Close the tab and never come back? Nobody's listening. After about 30 seconds, cancel it. Save the partial reply. Free the memory.

The GPUs are the cost of the whole business. So number the tokens, keep a log, and never pay the GPU twice.

Full breakdown in the lecture.