To make an iPhone stream feel live, throw frames away

transcript

The iOS Simulator only runs on a Mac. So I built a tool to stream it anywhere. The trick that made it feel live? Throwing frames away.

Version one: take a screenshot, send it down a WebSocket. Repeat. Starting the screenshot tool costs about 50 milliseconds. Encode the PNG: about 30. Base64 it for JSON: about 20. About 100 milliseconds a frame. 10 frames a second, at best. In practice, a few.

But here's the catch. Over a WebSocket, frames queue up behind each other. One slow frame, and every frame after it has to wait.

So I switched to WebRTC, with one rule. The frame queue holds exactly one frame. Slot empty? The new frame goes in. The stream picks it up. Slot still full? Most designs would wait, or grow the queue. This one throws the old frame away, and keeps the new one.

A stale picture of the screen is worth nothing. So the stream can skip frames. But it never falls behind.

Now the presets target 45 frames a second on a weak network. 60 in the middle. Up to 90 on a LAN. Real rates depend on the Mac and the network.

That's how a stream of screenshots feels like a device. For anything live, don't queue the past. The newest frame wins.

Full breakdown in the lecture.