Prime Video offers thousands of live streams. To catch problems such as frozen video, blocky corruption or sound out of sync with the picture, one team built a tool that watches the streams automatically and raises an alarm when it spots a defect.
The tool had three parts: a converter that turns a video stream into individual frames and audio chunks, detectors that inspect those frames for defects, and an orchestrator that runs the steps in order. The first version was built from serverless pieces, cloud services you pay for per use without running servers: AWS Step Functions for the orchestration, AWS Lambda for the detectors, and an Amazon S3 bucket to hold the frames between steps. It was quick to build.
In practice it hit a hard scaling limit at about 5% of the expected load, and it cost too much. Step Functions charges per state transition, and the tool made several transitions for every second of every stream, so it quickly hit account limits too. And every frame was uploaded to S3 and downloaded again by each detector, a huge number of paid storage requests.
The team's fix was to put all three parts into one process on regular EC2 and ECS servers. Frames now passed between parts in memory, so the S3 bucket wasn't needed, and orchestration became simple code in the same program. When one copy outgrew a machine, they ran several, each with different detectors.
The result: infrastructure cost down by over 90%, and enough capacity to monitor every stream customers watch, not just the most popular ones.
- Hit a hard limit at about 5% of expected load
- Over 90% lower infrastructure cost after the change
- Monitoring went from the top streams to all streams viewed
Why it matters: Every hop between services costs time and, in the cloud, money. When parts trade huge amounts of data constantly, one process can be the better design; the post says the choice "has to be made on a case-by-case basis".