YouTube Video Platform.

How Do You Turn Twenty Million Uploads a Day Into Playable Video, and Count Every View Honestly?

It is 6 pm in Mumbai and Asha has just finished editing a 12-minute cooking video on her phone. She taps "Upload" on a crowded local train. The file is 1.8 GB. Two stations later the train goes into a cutting and her phone drops to no signal.

Asha expects three things. The upload carries on where it stopped, not from zero. Her video is watchable a few minutes after the last byte lands, even if the 4K version takes longer. And when her audience starts watching, the view count on her video goes up with every real person, and not with every bot somebody points at it.

Twenty million other people will upload something today too.

This is the system we'll design today.

The Interview Begins#

You're interviewing at a company that runs a video platform. The interviewer pulls up a dashboard: a line climbing through the evening labelled "uploads in progress", and next to it a counter of views ticking too fast to read.

Your Questions#

Functional Requirements#

Non-Functional Requirements#

Upload Estimation#

Storage Estimation#

Transcoding Estimation#

Views and Recommendations Estimation#

Key Metrics Summary#

System Architecture#

%% title: Video platform architecture: upload, processing, views, recommendations
flowchart TD
  subgraph L1["Layer 1 · Clients"]
    direction LR
    CR@{ icon: "sd:phone", label: "Creators<br/>phone, browser, studio apps" }
    VW@{ icon: "sd:users", label: "Viewers<br/>web, apps, TVs" }
  end
  subgraph L2["Layer 2 · Edge"]
    direction LR
    UP@{ icon: "sd:gateway", label: "Upload front ends<br/>resumable sessions" }
    API@{ icon: "sd:api", label: "Watch and feed API" }
    CDN@{ icon: "sd:cdn", label: "CDN<br/>segments and manifests" }
  end
  subgraph L3["Layer 3 · Processing"]
    direction LR
    ORCH["Processing orchestrator<br/>one task graph per video"]:::hot
    TR@{ icon: "sd:cpu", label: "Transcode pool<br/>chunks in parallel" }
    CHK@{ icon: "sd:shield", label: "Checks<br/>policy, copyright, thumbnails" }
  end
  subgraph L4["Layer 4 · Storage"]
    direction LR
    ORIG@{ icon: "sd:object-storage", label: "Originals<br/>kept forever" }
    REND@{ icon: "sd:object-storage", label: "Renditions<br/>a rebuildable cache" }
    META@{ icon: "tech:mysql", label: "Video metadata<br/>sharded MySQL" }
  end
  subgraph L5["Layer 5 · Views and discovery"]
    direction LR
    EV@{ icon: "tech:kafka", label: "Kafka · view events" }
    CNT["View counter<br/>dedupe, filter, sharded counts"]:::core
    REC["Recommendations<br/>candidates, then ranking"]:::core
  end
  CR ==>|"chunks, resumable"| UP
  UP ==> ORIG
  ORIG -.->|"upload complete"| ORCH
  ORCH --> TR
  ORCH --> CHK
  TR ==> REND
  ORCH -->|"renditions ready"| META
  REND --> CDN
  VW ==> API
  VW ==>|"segments"| CDN
  API --> META
  API --> REC
  VW -.->|"view and watch-time events"| EV
  EV --> CNT
  EV --> REC
  class CR,VW client
  class UP,API,CDN edge
  class TR,CHK service
  class ORIG,REND,META store
  class EV queue

KEY INSIGHTS:

• The original is the source of truth; every rendition can be rebuilt from it, so processing can be retried, reordered and redone
• Processing is a task graph, not a function call: hundreds of chunks per video, scheduled across a shared pool with priorities
• The watch API only lists what is ready, so a video goes live in low quality and gains renditions as they finish
• Views and recommendations are event pipelines fed from the same stream

Data Flow: One Upload, From the Train to the First View#

%% title: Asha's upload, from her phone to the first counted view
sequenceDiagram
  autonumber
  participant A as Asha's app
  participant U as Upload front end
  participant S as Originals store
  participant O as Orchestrator
  participant T as Transcode pool
  participant V as Viewer
  A->>U: Start session, 1.8 GB, title, privacy
  U-->>A: Session URI
  A->>U: Chunks of 8 MB
  U->>S: Write part, then record the stored offset
  Note over A,U: train enters a cutting, connection drops at 1.1 GB
  A->>U: Status? bytes */1,800,000,000
  U-->>A: 308 · Range bytes=0-1,099,999,999
  A->>U: Remaining 700 MB from byte 1,100,000,000
  U->>S: Last part, checksum matches
  S-)O: Upload complete, video v_7Hq2
  O->>T: 180 chunks · 144p to 480p first
  T-->>O: Low ladder ready
  O->>O: Publish manifest with 144p to 480p
  V->>V: Plays at 480p, view event sent
  O->>T: 720p, 1080p, newer codec

MEANWHILE, IN PARALLEL:

• Policy and copyright checks run against the original as soon as it is complete; a video that fails them never gets a public manifest
• Thumbnails are generated from the first chunks, so Asha can choose one while the HD renditions are still being made

Deep Dive 1: Resumable Uploads (Week 2 Concepts: Idempotency, Timeouts)#

This builds on Week 2, Day 2 · Idempotency in Practice: the same request sent twice must have the effect of one.

The Solution:

YouTube's public API documents exactly this shape: a request starts a session and returns its URI in a Location header; the client sends the file in chunks that "must be a multiple of 256 KB"; to check progress it sends Content-Range: bytes */<size>; and an incomplete upload answers 308 Resume Incomplete with a Range header that says "how many bytes of the file have already been successfully uploaded".

%% title: The life of one upload session
stateDiagram-v2
  [*] --> OPEN: session created, URI returned
  OPEN --> RECEIVING: first chunk
  RECEIVING --> RECEIVING: chunk stored, offset advanced
  RECEIVING --> PAUSED: connection lost
  PAUSED --> RECEIVING: status query, resume from offset
  PAUSED --> EXPIRED: no activity for days
  RECEIVING --> COMPLETE: last byte stored, checksum ok
  RECEIVING --> FAILED: checksum mismatch
  COMPLETE --> [*]
  EXPIRED --> [*]
  FAILED --> [*]
  OPEN: OPEN · metadata saved, no bytes yet
  PAUSED: PAUSED · stored parts kept
  COMPLETE: COMPLETE · original sealed, processing starts
  class RECEIVING core
  class COMPLETE ok
  class FAILED bad
  class EXPIRED bad

• The front end stores parts, not files. Each chunk becomes a part in object storage (a multipart upload); the session row records which parts are sealed. Any front end can serve the next chunk, because the state lives in the session table, not in a socket.
• The offset only moves after the part is durable. That single rule makes resume correct.
• Sessions outlive connections, not forever. The API notes that each session URI "has a finite lifetime and eventually expires"; parts of expired sessions are garbage-collected.
• Metadata goes in first. Title, privacy and description are saved when the session opens, so the video exists (as a draft) even while bytes are still arriving.

# upload/session.py

"""
A resumable upload session (illustrative). The client sends chunks at any
offset; the server only advances `stored` after a part is durably written,
and answers status queries with that number. Resending bytes it already
has is harmless.
"""

import hashlib
from dataclasses import dataclass, field
from typing import Dict, Optional

CHUNK = 256 * 1024              # chunks are multiples of 256 KB, except the last


@dataclass
class Session:
    session_id: str
    video_id: str
    size: int                   # declared total size in bytes
    stored: int = 0             # bytes durably written, from offset 0
    parts: Dict[int, str] = field(default_factory=dict)   # part number -> etag
    state: str = "OPEN"


class UploadFrontEnd:
    def __init__(self, sessions, blobs, events):
        self.sessions, self.blobs, self.events = sessions, blobs, events

    def put(self, sid: str, offset: int, data: bytes) -> dict:
        s: Session = self.sessions.get(sid)
        if s.state == "EXPIRED":
            return {"status": 410}
        if s.state in ("COMPLETE", "FAILED"):
            return {"status": 200 if s.state == "COMPLETE" else 400}
        if offset != s.stored:
            return self.status(sid)                 # a gap, or bytes we already have: say where to resume
        last = offset + len(data) == s.size
        if len(data) % CHUNK and not last:
            return {"status": 400, "error": "chunk must be a multiple of 256 KB"}
        part_no = len(s.parts) + 1
        etag = self.blobs.put_part(s.video_id, part_no, data)    # durable before we move on
        s.parts[part_no] = etag
        s.stored += len(data)
        s.state = "RECEIVING"
        self.sessions.save(s)                       # conditional on the previous `stored`
        return self.status(sid)

    def status(self, sid: str) -> dict:
        s: Session = self.sessions.get(sid)
        if s.stored < s.size:
            return {"status": 308, "range": f"bytes=0-{s.stored - 1}" if s.stored else None}
        return {"status": 200}

    def finish(self, sid: str, client_sha256: Optional[str]) -> dict:
        s: Session = self.sessions.get(sid)
        if s.stored != s.size:
            return self.status(sid)
        sha = self.blobs.complete(s.video_id, s.parts)          # stitches parts, returns digest
        if client_sha256 and client_sha256 != sha:
            s.state = "FAILED"
            self.sessions.save(s)
            return {"status": 400, "error": "checksum mismatch"}
        s.state = "COMPLETE"
        self.sessions.save(s)
        self.events.publish("upload.complete", {"video_id": s.video_id, "sha256": sha})
        return {"status": 200, "video_id": s.video_id}


def sha256_of(path: str) -> str:
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for block in iter(lambda: f.read(1 << 20), b""):
            h.update(block)
    return h.hexdigest()

Design choices worth saying out loud:

• Chunk size is a trade-off. Bigger chunks mean fewer requests but more bytes resent after a drop. 8 MB on a phone, larger from a studio.
• Upload close to the creator, store the original once. The front end nearest Asha takes the bytes; the original is replicated by the storage layer, not by the client uploading twice.
• Don't start transcoding on a half-uploaded MP4. Many phone files put the index (the moov box) at the end, so the file can't be read until the last byte. A client that records fragmented MP4 lets processing start early; otherwise the pipeline starts on "upload complete".

Deep Dive 2: The Processing Pipeline as a Task Graph (Week 3 Concepts: Queues, Backpressure, Dead Letters)#

This builds on Week 3, Day 3 · Backpressure and Flow Control and Week 3, Day 4 · Dead Letters and Poison Pills.

The Challenge:

The Solution:

%% title: One video's processing task graph
flowchart TD
  IN(["upload.complete<br/>v_7Hq2, 12 min, 1080p source"]):::client
  PROBE["Probe<br/>codec, length, rotation, GOPs"]:::service
  SPLIT["Split at closed GOPs<br/>180 chunks of ~4 s"]:::core
  subgraph T1["Tier 1 · high priority"]
    direction LR
    LOW["Per chunk: decode once<br/>encode 144p to 480p"]:::hot
  end
  subgraph T2["Tier 2 · normal priority"]
    direction LR
    HD["Per chunk: 720p, 1080p"]:::core
  end
  subgraph T3["Tier 3 · by popularity, low priority"]
    direction LR
    ADV["Newer codec, 4K<br/>only if watched enough"]:::queue
  end
  CHECKS["Policy, copyright,<br/>thumbnails"]:::service
  ASM["Assemble per rendition<br/>duration must match"]:::service
  PUB(["Manifest updated<br/>tier by tier"]):::ok
  IN --> PROBE --> SPLIT
  PROBE --> CHECKS
  SPLIT --> LOW --> ASM
  SPLIT --> HD --> ASM
  SPLIT -.-> ADV -.-> ASM
  CHECKS -->|"must pass before public"| PUB
  ASM --> PUB

• Every task has a deterministic output key: (video, chunk, tier, encoder version). A retried task overwrites the same object with the same bytes, so retries are idempotent and a crash costs one chunk, not one video.
• Three queues, not one: tier 1 is latency-sensitive and never waits behind tiers 2 and 3; tier 3 fills the troughs of the daily cycle.
• Poison chunks go to a dead-letter queue after a few failures, with the error. Retrying a corrupt chunk forever wastes the pool; one bad chunk marks that rendition failed while the others publish.
• Integrity checks are cheap and essential: the Argos paper describes checks such as "video length must match the input", because one faulty accelerator could corrupt many videos.

# processing/plan.py

"""
Plan one video's processing as a task graph (illustrative). Each task is a
(chunk, tier) pair with a deterministic output key, so retries overwrite the
same object. Tiers are published independently: the video goes live with
tier 1 and gains tiers as their last chunk finishes.
"""

from dataclasses import dataclass
from typing import List

TIERS = {
    1: {"renditions": ["144p", "240p", "360p", "480p"], "priority": "high"},
    2: {"renditions": ["720p", "1080p"], "priority": "normal"},
    3: {"renditions": ["1440p", "2160p", "newer-codec ladder"], "priority": "low"},
}
CHUNK_S = 4.0
MAX_WORKERS_PER_VIDEO = 2_000
ENCODER = "enc-2026.10"


@dataclass(frozen=True)
class Task:
    video_id: str
    chunk: int
    start_s: float
    end_s: float
    tier: int

    @property
    def output_key(self) -> str:                      # same task, same key: retries are harmless
        return f"{self.video_id}/{ENCODER}/t{self.tier}/{self.chunk:05d}"


def tiers_for(height: int, popularity: str) -> List[int]:
    tiers = [1]
    if height >= 720:
        tiers.append(2)
    if height >= 1440 or popularity in ("hot", "warm"):
        tiers.append(3)                              # long-tail videos skip it until they earn it
    return tiers


def plan(video_id: str, duration_s: float, gop_starts: List[float], height: int, popularity: str):
    cuts, last = [0.0], 0.0
    for g in gop_starts:                             # cut only where a chunk decodes on its own
        if g - last >= CHUNK_S:
            cuts.append(g)
            last = g
    cuts.append(duration_s)
    chunks = list(zip(cuts, cuts[1:]))
    tasks = [Task(video_id, i, a, b, tier)
             for tier in tiers_for(height, popularity)
             for i, (a, b) in enumerate(chunks)]
    return {"tasks": tasks, "chunks": len(chunks), "cap": MAX_WORKERS_PER_VIDEO}


def on_task_done(state, task: Task) -> None:
    done = state.mark_done(task)                     # returns chunks finished for this tier
    if done == state.chunk_count(task.video_id):
        state.enqueue_assemble(task.video_id, task.tier)    # stitch, check duration, write manifest

Not every video deserves every rendition#

The Argos paper describes video popularity as "a stretched power law distribution" with three buckets: the very popular videos that make up most watch time (worth extra processing to cut bandwidth), modestly watched videos, and the long tail, "the majority of videos", where it makes sense "to minimize storage and transcoding costs while maintaining playability". It adds that "old videos can increase in popularity and may need to be reprocessed".

Deep Dive 3: Counting Views (Week 8 Concepts: Event Ingestion, Late Data; Week 1: Hot Keys)#

This builds on Week 8, Day 1 · Event Ingestion at Scale and Week 1, Day 4 · Hot Keys and Skew.

The Challenge:

The Solution:

%% title: From a play to the public count
flowchart TD
  P(["Player<br/>view_id per playback"]):::client
  ED@{ icon: "sd:gateway", label: "Event edge<br/>auth, rate limits" }
  K@{ icon: "tech:kafka", label: "Kafka · view events<br/>partitioned by view_id" }
  AGG@{ icon: "tech:flink", label: "Flink · dedupe by view_id,<br/>1-second totals per video" }
  SPAM["Quality filter<br/>address, device, tabs, progress"]:::core
  SH@{ icon: "tech:redis", label: "Redis · provisional count<br/>64 shards per hot video" }
  VER[("Verified counts<br/>public, corrections applied")]:::store
  DASH(["Creator analytics<br/>watch time, retention"]):::ok
  P --> ED --> K --> AGG
  AGG -->|"provisional"| SH
  AGG --> SPAM
  SPAM -->|"cleared views"| VER
  SPAM -.->|"invalid: subtract later"| VER
  VER --> DASH
  class ED edge
  class K queue
  class AGG service
  class SH cache

• Partition by playback, aggregate by video. Partitioning the stream by video_id would put a viral video on one partition; partitioning by view_id spreads it, and each worker sends one small total per video per second.
• Shard only the hot ones. A video above ~1,000 views a second gets 64 counter shards; reads sum them (cached for a second). Most videos keep one row.
• The public number is a read model built from verified counts, cached at the edge for a few seconds; nobody reads the raw shards.

# views/counter.py

"""
View counting (illustrative). Events are deduplicated by view_id within a
window, aggregated per video per second, and added to a sharded counter.
The public count only advances by views the quality filter has cleared.
"""

import time
import zlib
from collections import defaultdict
from typing import Dict

DEDUPE_TTL_S = 6 * 3600          # a retried event within 6 hours is a duplicate
HOT_SHARDS = 64
HOT_THRESHOLD = 1_000            # views/s above which a video's counter is sharded


class WindowAggregator:
    """One stream worker: its partitions are keyed by view_id, not video_id."""

    def __init__(self, seen, counters):
        self.seen, self.counters = seen, counters        # seen: TTL set; counters: sharded store
        self.window: Dict[str, int] = defaultdict(int)
        self.window_start = time.monotonic()

    def on_event(self, ev: dict) -> None:
        if ev["type"] != "view_start":
            return
        if not self.seen.add_if_absent(ev["view_id"], ttl=DEDUPE_TTL_S):
            return                                       # player retry: already counted
        self.window[ev["video_id"]] += 1
        if time.monotonic() - self.window_start >= 1.0:
            self.flush()

    def flush(self) -> None:
        for video_id, n in self.window.items():
            self.counters.add(video_id, n)               # one write per video per second per worker
        self.window.clear()
        self.window_start = time.monotonic()


class ShardedCounter:
    def __init__(self, kv, rates):
        self.kv, self.rates = kv, rates

    def add(self, video_id: str, n: int) -> None:
        if self.rates.views_per_s(video_id) < HOT_THRESHOLD:
            self.kv.incr(f"v:{video_id}", n)
            return
        shard = zlib.crc32(f"{video_id}:{time.time_ns()}".encode()) % HOT_SHARDS
        self.kv.incr(f"v:{video_id}:{shard}", n)         # 64 rows share the heat

    def read(self, video_id: str) -> int:
        keys = [f"v:{video_id}"] + [f"v:{video_id}:{s}" for s in range(HOT_SHARDS)]
        return sum(self.kv.mget(keys, default=0))        # 64-bit integers, never 32

Provisional, verified and engaged#

Deep Dive 4: Recommendations in 100 ms (Week 7 Concepts: Retrieval and Relevance; Week 4: Caching)#

This builds on Week 7, Day 3 · Query Processing and Relevance: retrieve broadly and cheaply, then rank precisely.

What YouTube has published: the 2016 paper describes "the classic two-stage information retrieval dichotomy": a candidate generation network that takes events from the viewer's history and retrieves "a small subset (hundreds) of videos from a large corpus", then a ranking network that assigns each a score; at serving time "an approximate nearest neighbor lookup is performed to generate hundreds of candidate video" recommendations. Ranking optimises "expected watch time per impression" using weighted logistic regression, because ranking by click-through rate "promotes deceptive videos that the user does not complete". An "example age" feature lets the model learn that viewers prefer fresh videos. YouTube's 2021 post on the system lists its signals as "clicks, watchtime, survey responses, sharing, likes, and dislikes", and says that when it first weighted watch time in 2012, views dropped by 20%.

%% title: Serving one recommendation request
flowchart TD
  REQ(["Rohan opens Home<br/>user id, device, time"]):::client
  FS@{ icon: "sd:database", label: "Feature store<br/>recent watches, searches" }
  subgraph CG["Stage 1 · candidates, ~5 ms each, in parallel"]
    direction LR
    ANN@{ icon: "sd:index", label: "ANN index<br/>video embeddings" }
    SUB["Subscriptions<br/>new uploads"]:::service
    COW["Co-watch<br/>after what he just watched"]:::service
    FRESH["Fresh and trending<br/>in his language"]:::service
  end
  MERGE["Merge, dedupe<br/>~500 candidates"]:::core
  RANK@{ icon: "sd:ml", label: "Ranking model<br/>expected watch time, satisfaction" }
  RULES["Rules<br/>watched, diversity, policy"]:::core
  OUT(["20 videos<br/>logged as impressions"]):::ok
  REQ --> FS --> CG
  ANN --> MERGE
  SUB --> MERGE
  COW --> MERGE
  FRESH --> MERGE
  MERGE --> RANK --> RULES --> OUT
  class FS store
  class ANN cache
  class RANK service

# recs/serve.py

"""
Two-stage recommendation serving (illustrative). Candidate sources are cheap
and run in parallel; only their union is scored by the ranking model. The
response is logged as impressions, which become training labels when the
view events come back.
"""

import asyncio
from typing import List

SOURCES_TIMEOUT_S = 0.020
MAX_CANDIDATES = 500
SHOW = 20
MAX_PER_CHANNEL = 2


async def recommend(user_id: str, ctx: dict, deps) -> List[dict]:
    feats = await deps.features.get(user_id)                 # recent watches, searches, locale
    user_vec = deps.user_tower(feats, ctx)                   # one embedding per request

    sources = [
        deps.ann.search(user_vec, k=300),                    # nearest video embeddings
        deps.subscriptions.new_uploads(user_id, k=100),
        deps.cowatch.after(feats["last_watched"], k=100),
        deps.fresh.trending(feats["language"], k=50),
    ]
    done = await asyncio.wait_for(asyncio.gather(*sources, return_exceptions=True),
                                  timeout=SOURCES_TIMEOUT_S)
    seen, cands = set(feats["watched_recently"]), []
    for result in done:
        if isinstance(result, Exception):
            continue                                         # a slow source is skipped, not awaited
        for v in result:
            if v["id"] not in seen:
                seen.add(v["id"])
                cands.append(v)
    cands = cands[:MAX_CANDIDATES]

    scores = await deps.ranker.score(feats, ctx, cands)      # batched on accelerators
    ranked = sorted(zip(cands, scores), key=lambda p: p[1]["value"], reverse=True)

    out, per_channel = [], {}
    for v, s in ranked:
        if s["borderline"] or per_channel.get(v["channel"], 0) >= MAX_PER_CHANNEL:
            continue
        per_channel[v["channel"]] = per_channel.get(v["channel"], 0) + 1
        out.append({"video_id": v["id"], "score": s["value"]})
        if len(out) == SHOW:
            break
    deps.impressions.log(user_id, ctx, out)                  # labels arrive later as view events
    return out


def value(p_click: float, expected_watch_s: float, p_satisfied: float) -> float:
    """What the ranker optimises: watch time a viewer would value, not raw clicks."""
    return p_click * expected_watch_s * (0.5 + p_satisfied)

Caching, carefully: the home page for a returning viewer can be precomputed in the background and refreshed on open, but Up next depends on the video just watched and must be computed live. A new upload has no co-watch history; the subscriptions and fresh sources, plus the example-age feature, give it a chance to be seen.

A Video's First Day#

Edge Cases#

Monitoring Dashboard#

SLOs for the Platform#

The interviewer leans back. "One more."

Summary: Concepts Applied from 10-Week Course#

Why YouTube Matters#

Self-Assessment Checklist#

After studying this case study, you should be able to:

Upload and processing:

Views:

Recommendations and operations:

Sources#

YouTube (official):

• Talk

  YouTube — Happy 20th birthday, YouTube (20 billion+ videos uploaded, 20 million+ uploaded daily on average, 100 million+ comments a day in 2024), April 2025: https://blog.youtube/news-and-events/happy-birthday-youtube-20/

• Talk

  YouTube — Reimagining video infrastructure to empower YouTube (the VCU, "up to 20-33x improvements in compute efficiency", "more than 500 hours of video content on average is uploaded every minute"), April 2021: https://blog.youtube/inside-youtube/new-era-video-infrastructure/

• Talk

  YouTube Official Blog — press page (200 billion+ daily Shorts views; localised in 100+ countries and 80 languages), as of October 2026: https://blog.youtube/press/

• Talk

  Cristos Goodrow — On YouTube's recommendation system (signals; valued watchtime; the 2012 watch-time change and 20% drop in views), September 2021: https://blog.youtube/inside-youtube/on-youtubes-recommendation-system/

YouTube help centre and developer docs (as of October 2026):

• Talk

  How engagement metrics are counted (views counted from the first frame from 24 August 2026; counts may be slowed, frozen or changed; low-quality playbacks): https://support.google.com/youtube/answer/2991785

• Talk

  Low video quality after upload (low quality first; 4K 60-minute example up to 4 hours): https://support.google.com/youtube/answer/71674

• Talk

  Upload videos longer than 15 minutes (256 GB or 12 hours): https://support.google.com/youtube/answer/71673

• Talk

  YouTube Data API — Resumable uploads (session URI, 256 KB chunks, 308 Resume Incomplete, Range header): https://developers.google.com/youtube/v3/guides/using_resumable_upload_protocol

Papers:

• Talk

  Parthasarathy Ranganathan et al. — Warehouse-scale video acceleration: co-design and deployment in the wild, ASPLOS 2021 (task dependency graph, global work queue, closed-GOP chunks in parallel, multiple-output transcoding, popularity buckets, integrity checks): https://research.google/pubs/pub50300/

• Talk

  Paul Covington, Jay Adams, Emre Sargin — Deep Neural Networks for YouTube Recommendations, RecSys 2016 (two stages, hundreds of candidates, ANN at serving, expected watch time, example age): https://research.google/pubs/pub45530/

Company filings and press:

• Paper

  Alphabet — Q4 and fiscal year 2025 results (YouTube revenue above $60 billion across ads and subscriptions), February 2026: https://s206.q4cdn.com/479360582/files/doc_financials/2025/q4/2025q4-alphabet-earnings-release.pdf

• Talk

  Tubefilter — Alphabet reveals YouTube revenue for first time, February 2026: https://tubefilter.com/2026/02/05/alphabet-q4-2025-youtube-earnings-revenue-60-billion/

• Talk

  PPC Land — YouTube counts views from the first frame across all formats on August 24 (citing TeamYouTube's announcement of 17 August 2026): https://ppc.land/youtube-counts-views-from-the-first-frame-across-all-formats-on-august-24/

• Talk

  Tubefilter — YouTube is making Shorts views go up (Shorts views count every start or replay from 31 March 2025), March 2025: https://www.tubefilter.com/2025/03/26/youtube-shorts-views-counting-stats/

• Talk

  BBC News — Gangnam Style music video 'broke' YouTube view limit (YouTube's 32-bit joke; the counter already moved to 64 bits), December 2014: https://www.bbc.com/news/world-asia-30288542

Other engineering references:

• Talk

  Vitess — History ("created in 2010 to solve the MySQL scalability challenges that the team at YouTube faced"): https://vitess.io/docs/overview/history/

• Book

  Ryan Peterman, Haixia Shi (Meta) — Reducing Instagram's basic video compute time by 94 percent (86.17 CPU-seconds for a 23-second 720p x264 transcode), November 2022: https://engineering.fb.com/2022/11/04/video-engineering/instagram-video-processing-encoding-reduction/

Design note: the architecture, code, thresholds, tiers, rates, sizes and dashboards here are an illustrative design for the problem, not a description of YouTube's internal systems beyond what YouTube and Google have published.

Further Reading#

Related systems to study:

• Read

  Netflix (Bonus Problem 6 in this series): the delivery half: adaptive bitrate, per-title encoding and a CDN inside ISPs

• Talk

  Instagram (Bonus Problem 21): feed ranking, fan-out and media for photos, videos and stories

• Read

  Google Search (Bonus Problem 5): retrieval and ranking over an index of billions

• Read

  Twitter / X (Bonus Problem 10): hot keys and fan-out at scale

Foundational concepts (this course):

• Read

  Week 2, Day 2 — Idempotency in Practice

• Read

  Week 3, Day 3 — Backpressure and Flow Control

• Read

  Week 3, Day 4 — Dead Letters and Poison Pills

• Read

  Week 1, Day 4 — Hot Keys and Skew

• Read

  Week 8, Day 1 — Event Ingestion at Scale

• Read

  Week 7, Day 3 — Query Processing and Relevance

"The upload is the only thing you can't rebuild. Everything after it, renditions, counts and recommendations, is a pipeline you can run again."