GitHub.

How Do You Keep a Billion Repositories Identical on Five Machines Each, While People and Agents Push to Them Around the Clock?

It is a Thursday morning in late October 2026 in Bengaluru. Nandini Hegde maintains an open-source Python library that parses and validates Indian postal addresses and PIN codes. Thousands of projects depend on it. Today is the 4.0 release. At 10:40 she adds the last pull request to the merge queue, which tests it against the newest main together with the two pull requests ahead of it. At 11:02, from her laptop in Koramangala, she runs git push origin v4.0.0 over SSH.

That one tag is a small write with a long tail. It must reach a majority of the repository's replicas before GitHub says yes. It wakes the release workflow: a matrix of 24 jobs on Linux, macOS and Windows, each on a fresh virtual machine, building packages from code anyone could have proposed. It sends webhooks to her documentation host and to the project's chat bot. Within the hour, thousands of other projects' CI pipelines fetch her repository to test against the new version. Her co-maintainer in Pune has three coding agents pushing to their own branches of the same repository every few minutes. And at 11:20 someone searches all of GitHub for parse_pin(, the function she deprecated, to see whose code will break.

This is the system we'll design today.

The Interview Begins#

You're interviewing at a company that builds developer tools. The interviewer pastes the end of a terminal session into the shared document: * [new tag] v4.0.0 -> v4.0.0. Next to it, a screenshot of a repository's Actions tab: "Release · 24 jobs · Queued". Under it, a webhook's "Recent deliveries" list, one row red: "Failed · timed out".

Your Questions#

Functional Requirements#

Non-Functional Requirements#

Git Traffic#

Storage#

Actions#

Webhooks and Events#

Key Metrics Summary#

System Architecture#

%% title: A code host: vote on the write, fan out the event, run CI on disposable machines
flowchart TD
  subgraph L1["Layer 1 · Clients"]
    direction LR
    DEV@{ icon: "sd:laptop", label: "Developers and agents<br/>git over SSH or HTTPS" }
    WEB@{ icon: "sd:browser", label: "Web and API<br/>pull requests, search" }
  end
  subgraph L2["Layer 2 · Edge"]
    direction LR
    LB@{ icon: "sd:load-balancer", label: "GLB, layer 4<br/>one IP, many proxies" }
    GITP["Git proxies<br/>SSH and HTTPS, auth"]:::service
    APP["Rails monolith<br/>web, API, PRs"]:::service
  end
  subgraph L3["Layer 3 · Git storage"]
    direction LR
    SPK["Spokes<br/>route, vote, repair"]:::hot
    FS@{ icon: "sd:servers", label: "Fileservers<br/>five replicas, several sites" }
    BLOB@{ icon: "sd:object-storage", label: "Azure Blob Storage<br/>virtual replica" }
  end
  subgraph L4["Layer 4 · Metadata and events"]
    direction LR
    MY@{ icon: "tech:mysql", label: "MySQL and Vitess<br/>PRs, issues, users" }
    K@{ icon: "tech:kafka", label: "Kafka<br/>one event per push" }
    JOBS@{ icon: "sd:worker", label: "Background jobs<br/>60+ push tasks" }
  end
  subgraph L5["Layer 5 · Outward"]
    direction LR
    HOOK@{ icon: "sd:webhook", label: "Webhook delivery<br/>per-endpoint queues" }
    ACT["Actions control plane<br/>queues, fair share"]:::core
    RUN@{ icon: "sd:server", label: "Runner VMs<br/>one job each" }
    SRCH@{ icon: "sd:search", label: "Code search<br/>sharded by blob" }
  end
  DEV ==> LB
  WEB ==> LB
  LB ==> GITP
  LB --> APP
  GITP ==> SPK
  APP --> SPK
  APP --> MY
  SPK ==> FS
  SPK -.->|"durable copy"| BLOB
  SPK ==>|"ref updated"| K
  K ==> JOBS
  JOBS ==> HOOK
  JOBS ==> ACT
  ACT ==> RUN
  RUN -.->|"clone"| GITP
  K --> SRCH
  class DEV,WEB client
  class LB edge
  class FS,BLOB,MY store
  class K queue
  class JOBS,RUN,HOOK,SRCH service

KEY INSIGHTS:

• Git data and metadata live in different systems. Repositories are files on fileservers replicated by Spokes; pull requests, issues, users and permissions are rows in MySQL. They fail differently and are replicated differently
• The write is a vote, the rest is an event. Only the reference update needs agreement among replicas. Everything that follows (pull requests, webhooks, CI, search) reads from one event and can lag, retry or fail on its own
• Strangers' code runs on machines that are thrown away. The compute plane is multi-tenant by design, so isolation is per job, not per customer

Data Flow: Nandini Pushes the Tag#

%% title: One tag push, from Nandini's laptop to a majority of replicas and onwards
sequenceDiagram
  autonumber
  participant N as Nandini's laptop
  participant P as Git proxy, SSH
  participant S as Spokes
  participant R as Five replicas
  participant K as Kafka
  participant J as Push jobs
  N->>P: git push over SSH, key checked
  P->>S: Which replicas hold this repository
  S-->>P: Five replicas, preference order
  P->>R: Send pack, new objects only
  S->>R: Vote on ref update, old and new value
  R-->>S: Locked, applied, checksums match on 5 of 5
  S->>R: Commit
  S-->>P: Committed on a majority
  P-->>N: new tag v4.0.0
  S->>K: Push event, repo, ref, old and new SHA
  K->>J: Fan out to independent consumers
  J->>J: PR sync, webhooks, workflow trigger, search

MEANWHILE, IN PARALLEL:

• The release workflow's 24 jobs enter the Actions queue of Nandini's organization; warm VMs pick them up and clone the tag through the same Git proxies
• Two webhook deliveries go into the queues for her documentation host and her chat bot; each is signed and carries a delivery id

Deep Dive 1: Storing a Billion Repositories (Week 1 Concepts: Replication Trade-offs; Week 5: Consistency Models, Leader Election)#

This builds on Week 1, Day 2 · Replication Trade-offs, Week 5, Day 1 · Consistency Models and Week 5, Day 5 · Leader Election.

What is published: GitHub announced DGit in April 2016: three copies of every repository on three servers, with a write "only committed if at least two replicas confirm success". In September 2016 Patrick Reynolds renamed it Spokes: copies sit in different racks; no update is accepted "unless a strict majority of the replicas can apply the change and get the same result"; every write "acquires an exclusive lock on a majority of replicas", so "Spokes eliminates conflicts by eliminating concurrent writes entirely"; and rather than electing a primary, Spokes "treats every write as an election". Losers of a vote are marked unhealthy until repaired; on a typical day "a few dozen writes" aren't unanimous. In 2017 Michael Haggerty described a three-phase commit, with the replicas also used as a distributed lock, costing four round trips to distant replicas (60 to 80 ms each across the US) and about a third of a second per reference update, and a checksum per replica built from the XOR of a hash of every (reference, value) pair, so it can be updated incrementally.

%% title: One reference update as a vote across five replicas
flowchart TD
  P["Push arrives<br/>pack already on replicas"]:::service
  LOCK["Phase 1 · lock<br/>repository or reference on each replica"]:::core
  CHECK{"Old value matches<br/>and new value valid?"}
  APPLY["Phase 2 · apply tentatively<br/>compute new checksum"]:::core
  VOTE{"Strict majority<br/>with equal checksums?"}
  COMMIT(["Phase 3 · commit<br/>acknowledge the push"]):::ok
  ROLL(["Roll back everywhere<br/>reject the push"]):::bad
  REPAIR@{ icon: "sd:worker", label: "Sweeper<br/>repair the losers" }
  P ==> LOCK ==> CHECK
  CHECK ==>|"yes"| APPLY ==> VOTE
  CHECK -->|"no"| ROLL
  VOTE ==>|"yes"| COMMIT
  VOTE -->|"no"| ROLL
  COMMIT -.->|"minority disagreed"| REPAIR
  class REPAIR service

• Majority, not "two copies". Two copies survive one failure, but conflicting writes must be applied in the same order everywhere, and a majority lock makes concurrent writers impossible. With five replicas, a write needs three.
• Failure detection from real traffic. Spokes marks a fileserver offline after three requests fail in a row, and each application server keeps its own view, so one confused app server can't take a healthy fileserver out for everyone. Offline only moves a replica to the end of the read list.
• Repair is N-to-N, across racks. When a server dies, new copies of its repositories are made from any surviving replica onto any server, never two in one rack, so the bigger the fleet, the faster the repair. By 2021 GitHub kept five replicas "across our three data centers".

Compare: the database that did fail over#

The 43-second partition in October 2018 hit GitHub's MySQL clusters, which hold "non-Git metadata" such as pull requests and issues. Each had one primary, and Orchestrator promoted West Coast primaries during a 43-second network cut; the East still held a few seconds of writes the West never got, and service was degraded for 24 hours and 11 minutes. GitHub's first fix was to forbid promotions across regions. A single-primary design must decide where it may fail over to; a per-write majority never fails over, it just refuses writes when it can't reach a majority.

2026: one copy in another region, one in object storage#

What is published: In October 2026 GitHub wrote that a copy of 99.99999% of repositories now lives in Azure; most repositories hold two of their five replicas there and Azure serves about half of Git traffic, with reads sent to a replica in the same region and zone. But the quorum "waits for every healthy replica to respond (up to a timeout)" and each write "holds a repository-wide lock until it commits", so the slowest replica sets latency and throughput, and an Azure replica is about 25 ms away. GitHub's fix, now rolling out: change the quorum's scope "from the whole repository to individual references", since "Git's consistency requirements apply to each reference individually"; in internal testing with realistic traffic, writes to different references "consistently saw 3–4x higher throughput at lower latency". The next step makes Azure Blob Storage a member of the quorum, a "virtual replica", so a write isn't acknowledged until it's durable in object storage; read replicas then warm their caches from it without joining the quorum. In its 6 October post GitHub said its new architecture "has delivered up to 35 times higher write throughput" in internal benchmarks.

• Per-reference serialization keeps Git's guarantee (no reader sees a branch go back) while agents pushing to their own branches proceed in parallel; storage apart from compute fixes the old trap, where adding a replica for reads added a voter to every write.

# spokes/ref_update.py

"""
A reference update as a majority vote (illustrative, after Spokes' three-phase
commit), locked per reference as in GitHub's 2026 design. The checksum is the XOR
of hash(refname, value) over all references, so an update changes it by two hashes.
"""

import hashlib

def h(ref: str, sha: str) -> int:
    return int.from_bytes(hashlib.sha256(f"{ref}\0{sha}".encode()).digest()[:8], "big")


def new_checksum(old: int, ref: str, old_sha: str, new_sha: str) -> int:
    return old ^ h(ref, old_sha) ^ h(ref, new_sha)


def update_ref(replicas, ref: str, old_sha: str, new_sha: str, timeout_s: float = 0.5) -> bool:
    need = len(replicas) // 2 + 1                       # 3 of 5
    locked = [r for r in replicas if r.lock(ref, timeout_s)]
    if len(locked) < need:
        for r in locked: r.unlock(ref)
        return False                                    # no majority: refuse, never diverge
    votes = {}
    for r in locked:                                    # phase 2: prepare
        ok, checksum = r.prepare(ref, old_sha, new_sha)   # checks old value, applies tentatively
        if ok:
            votes.setdefault(checksum, []).append(r)
    winners = max(votes.values(), key=len, default=[])
    if len(winners) >= need:                            # phase 3: commit
        for r in winners: r.commit(ref)
        for r in set(locked) - set(winners): r.rollback(ref); r.mark_unhealthy()
        return True
    for r in locked: r.rollback(ref)
    return False

When one repository is the whole company#

What is published: In March 2021 GitHub described Project Cyclops, after its largest monorepo customers and its own github/github saw push failures. Maintenance runs "after every 50 git push operations, or after we receive 40MB of unpacked files"; limiting expensive delta comparisons in git repack "eliminated nearly all" maintenance failures; failed maintenance was retried every four hours instead of after seven days; an old per-fileserver push cap was removed; and replica checksums, which took 20 to 30 seconds inside the lock during repairs, were precomputed, cutting the lock "to under 1 second". In April 2021 Taylor Blau described geometric repacking with multi-pack reachability bitmaps, which cut average repack time "from around 1 minute... to just 15 seconds". Bitmaps let a server work out what to send by bitwise operations: walking the Linux kernel's objects fell from over 33 seconds to 1.57.

Deep Dive 2: The Front Door: Push, Clone and Fetch over SSH and HTTPS (Week 4 Concepts: Multi-Tier Caching; Week 1: Rate Limiting; Week 2: Timeouts)#

This builds on Week 4, Day 5 · Multi-Tier Caching, Week 1, Day 3 · Rate Limiting at Scale and Week 2, Day 1 · Timeout Hell.

What is published: GitHub's GLB (2018) "powers the majority of GitHub's public web and git traffic": a layer-4 load balancer with stateless directors that pick a primary and a secondary proxy per connection, so directors can change without breaking open connections. Spokes returns "up-to-date replicas, sorted in preference order" and routes reads "to the closest replica that is in sync". Before planned reboots, fileservers are quiesced: no new reads, but "existing read operations, including clones, are allowed to finish", which "can take anywhere from a few seconds to many hours". A 2020 GitHub post by Derrick Stolee recommends blobless clones for developers and long-lived build machines, and shallow clones only for builds deleted after one use. On 9 February 2026 a burst of cache rewrites caused "connection exhaustion in the service proxying Git operations over HTTPS"; HTTPS pushes and pulls failed while "SSH-based Git operations were not affected".

%% title: A fetch, from the edge to the nearest in-sync replica
flowchart TD
  C@{ icon: "sd:terminal", label: "git fetch<br/>SSH or HTTPS" }
  LB@{ icon: "sd:load-balancer", label: "GLB<br/>stateless directors" }
  PX["Git proxy<br/>auth, rate limits"]:::service
  RT{"Spokes route<br/>in-sync replicas, nearest first"}
  PC@{ icon: "sd:cache", label: "Pack cache<br/>hot commits, our design" }
  FS1@{ icon: "sd:server", label: "Replica, same zone<br/>bitmaps answer what to send" }
  FS2@{ icon: "sd:server", label: "Next replica<br/>after 3 failures in a row" }
  C ==> LB ==> PX ==> RT
  RT ==>|"popular tip"| PC
  RT ==>|"first choice"| FS1
  RT -.->|"failover"| FS2
  class C client
  class LB edge
  class PC cache
  class FS1,FS2 store

• Two protocols, two proxy fleets (our design). GitHub's February 2026 report names a separate "Git HTTPS proxy" layer, and SSH Git kept working. Here SSH and HTTPS share replica routing but run on separate proxy fleets, so one can fail without the other.
• Authenticate, then authorize per repository (read, write, protected branch) before Spokes is asked anything.
• Timeouts sized to the operation. A reference advertisement takes milliseconds; a clone can stream for an hour. One idle timeout for both either kills big clones or lets stuck connections pile up.
• Shed by cost. A full clone of a monorepo isn't a git ls-remote; when fileservers saturate, admit cheap operations and shed the most expensive first.

# gitproxy/route.py

"""
Picking a replica for a read (illustrative, after Spokes' published rules): in-sync
replicas nearest first; three failures in a row moves a replica to the end of the
list, not off it; a quiescing replica takes no new reads.
"""

FAILS_TO_DEMOTE = 3


def read_order(replicas: list, fail_streak: dict, draining: set) -> list:
    usable = [r for r in replicas if r.in_sync and r.name not in draining]
    healthy = [r for r in usable if fail_streak.get(r.name, 0) < FAILS_TO_DEMOTE]
    suspect = [r for r in usable if fail_streak.get(r.name, 0) >= FAILS_TO_DEMOTE]
    return sorted(healthy, key=lambda r: r.distance) + suspect    # try suspects last, not never


def serve_fetch(repo, replicas, fail_streak, draining, request):
    for r in read_order(replicas, fail_streak, draining):
        try:
            pack = r.upload_pack(repo, request, timeout_s=request.budget_s)
            fail_streak[r.name] = 0
            return pack
        except (ConnectionError, TimeoutError):
            fail_streak[r.name] = fail_streak.get(r.name, 0) + 1
    raise RuntimeError("no replica could serve the fetch")

Deep Dive 3: After the Push: Pull Requests, Test Merges and the Merge Queue (Week 3 Concepts: Queue vs Stream, Dead Letters; Week 1: Hot Keys)#

This builds on Week 3, Day 1 · Queue vs Stream, Week 3, Day 4 · Dead Letters and Poison Pills and Week 1, Day 4 · Hot Keys and Skew.

What is published: In June 2024 Will Haltom described how one huge background job, RepositoryPushJob, ran every piece of push processing in sequence. Retries were avoided because some steps tolerate duplicates and others don't ("it is also not desirable to dispatch multiples of the same webhook"), and an error in the Pushes MySQL cluster at the start could stop pull requests syncing. GitHub replaced it with a Kafka event per push, fanned out by independent consumers to jobs grouped by owner and retry needs, with a publisher "that would retry until broker acknowledgement". Ownership spread to 15+ teams, and fully processed pushes rose from about 99.897% to at least 99.999%. In 2017 GitHub explained that each push to a pull request branch creates "a test merge and test rebase", and a push to master does so "for every active pull request whose target is master", sometimes "more than a hundred references", keeping a repository busy for about 30 seconds; it coalesced these updates and gave user pushes priority.

%% title: One push event, many isolated consumers
flowchart TD
  S["Spokes<br/>ref updated"]:::hot
  K@{ icon: "tech:kafka", label: "Kafka<br/>push event, reliable publisher" }
  PR@{ icon: "sd:worker", label: "PR sync<br/>test merges, retry freely" }
  WH@{ icon: "sd:webhook", label: "Push webhooks<br/>at most once, time-bound" }
  WF@{ icon: "sd:worker", label: "Workflow trigger<br/>keyed by event id" }
  IDX@{ icon: "sd:index", label: "Search indexing<br/>new blobs only" }
  DB@{ icon: "tech:mysql", label: "Push records<br/>idempotent rows" }
  DLQ["Dead letters<br/>alert the owner"]:::bad
  S ==> K
  K ==> PR
  K ==> WH
  K ==> WF
  K --> IDX
  K --> DB
  PR -.->|"retries exhausted"| DLQ
  class K queue
  class PR,WF,IDX service
  class WH service
  class DB store

• Group by retry semantics. Tasks that tolerate duplicates retry for hours; must-happen-once tasks are keyed by the event id; time-bound tasks give up after a deadline.
• Each consumer keeps its own offset, so search indexing an hour behind doesn't delay pull request sync. A task that exhausts its retries goes to a dead-letter queue with an owner and an alert, not into a "catch everything" block.
• Merges contend on one reference. GitHub wrote in 2026 that merge queues and release trains "funnel all that work onto a single ref", and pull request merges grew to nearly 4 times their volume of a year earlier. A busy main is a hot key that per-reference locking can't split.
• The merge queue turns contention into batches. GitHub's docs describe temporary gh-readonly-queue/{base_branch} branches, a merge_group event for CI, and a build concurrency of 1 to 100 groups. GitHub's own monorepo merges about 2,500 pull requests a month through it, with a 33% shorter average wait (March 2024).
• Degrade, don't fail. By October 2026, half of sampled pull request traffic could "continue with limited functionality when a shared dependency fails".

# push/consumers.py

"""
Push follow-up work as independent consumers (illustrative): each task group has
its own retry policy; once-only tasks are keyed by the event id, and time-bound
tasks give up when the event is too old to be useful.
"""

import time

POLICIES = {
    "pr_sync":       {"max_attempts": 20, "deadline_s": 6 * 3600, "once": False},
    "push_webhooks": {"max_attempts": 3,  "deadline_s": 10 * 60,  "once": True},
    "workflows":     {"max_attempts": 10, "deadline_s": 3600,     "once": True},
    "push_records":  {"max_attempts": 50, "deadline_s": 24 * 3600, "once": False},
}


def handle(task: str, event: dict, attempt: int, done_keys, run, dead_letter, retry_later):
    p = POLICIES[task]
    key = f"{task}:{event['id']}"
    if p["once"] and done_keys.contains(key):
        return                                          # already done by an earlier attempt
    if time.time() - event["pushed_at"] > p["deadline_s"]:
        return dead_letter(task, event, reason="too old to be useful")
    try:
        run(task, event)
        if p["once"]:
            done_keys.add(key)
    except Exception as err:
        if attempt + 1 >= p["max_attempts"]:
            dead_letter(task, event, reason=repr(err))
        else:
            retry_later(task, event, attempt + 1, backoff_s=min(2 ** attempt, 300))

Deep Dive 4: Webhooks to the World (Week 2 Concepts: Webhook Delivery, Idempotency, Circuit Breakers)#

This builds on Week 2, Day 4 · Webhook Delivery System, Week 2, Day 2 · Idempotency in Practice and Week 2, Day 3 · Circuit Breakers.

What is published: GitHub's docs ask receivers to "respond with a 2XX response within 10 seconds", or "GitHub terminates the connection and considers the delivery a failure", and suggest a queue to process later. "GitHub does not automatically redeliver failed webhook deliveries"; receivers can redeliver from the past 3 days. The X-GitHub-Delivery GUID "will be the same as in the original delivery" on redelivery; X-Hub-Signature-256 is an HMAC-SHA256 of the body; payloads over 25 MB aren't delivered. After the 2018 partition, "over five million hook events" were queued; GitHub weighed recovery against "potentially overloading our ecosystem partners", and ~200,000 payloads "that had outlived an internal TTL" were dropped.

%% title: Deliveries isolated per destination
flowchart TD
  EV@{ icon: "tech:kafka", label: "Events<br/>push, PR, workflow" }
  FAN["Fan-out<br/>subscriptions, sign, delivery id"]:::core
  Q1@{ icon: "sd:queue", label: "Queue · docs host<br/>4 in flight" }
  Q2@{ icon: "sd:queue", label: "Queue · chat bot<br/>4 in flight" }
  Q3@{ icon: "sd:queue", label: "Queue · CI vendor<br/>breaker open" }
  SEND["Senders<br/>10 s timeout, log result"]:::service
  LOG@{ icon: "sd:log", label: "Delivery log<br/>3 days, redeliver API" }
  EXT@{ icon: "sd:globe", label: "Receivers<br/>verify, dedupe, queue" }
  EV ==> FAN
  FAN ==> Q1
  FAN ==> Q2
  FAN -.-> Q3
  Q1 ==> SEND
  Q2 ==> SEND
  SEND ==> EXT
  SEND --> LOG
  class EV queue
  class Q1,Q2,Q3 queue
  class LOG store
  class EXT external

• Per-destination queues and caps. A destination is a host, not a hook: thousands of repositories may point at one CI vendor, so cap in-flight deliveries per host.
• Breakers per host. After a run of timeouts or 5xx responses, stop sending, keep queueing for a bounded time, probe occasionally, close on success.
• A delivery has a shelf life, set per event type and recorded in the log when it expires. In 2018 an internal TTL dropped ~200,000 payloads until GitHub raised it: make the shelf life explicit, visible in the log and recoverable by redelivery.
• Order is not promised; ramp the recovery. After a retry, two events for one pull request can arrive in either order, so receivers fetch current state when order matters. A host that comes back is drained at a ramped rate.

# receiver/github_hook.py

"""
A receiver that does what GitHub's docs ask (illustrative): verify the signature,
answer within 10 seconds by queueing, and use X-GitHub-Delivery as an idempotency key.
"""

import hashlib
import hmac


def handle(headers: dict, body: bytes, secret: bytes, seen, queue) -> int:
    sig = headers.get("X-Hub-Signature-256", "")
    expected = "sha256=" + hmac.new(secret, body, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, sig):
        return 401                                      # forged or tampered
    delivery = headers["X-GitHub-Delivery"]             # same GUID on redelivery
    if not seen.add_if_absent(delivery, ttl_s=7 * 86400):
        return 200                                      # already handled: acknowledge, do nothing
    queue.put({"event": headers["X-GitHub-Event"], "delivery": delivery, "body": body})
    return 202                                          # well inside 10 s; the rebuild happens later

Deep Dive 5: GitHub Actions: Queueing, Runners, Isolation and Cost (Week 9 Concepts: Tenant Isolation, Noisy Neighbor; Week 3: Backpressure; Week 10: Capacity Planning)#

This builds on Week 9, Day 1 · Tenant Isolation, Week 9, Day 2 · Noisy Neighbor Prevention, Week 3, Day 3 · Backpressure and Flow Control and Week 10, Day 4 · Capacity Planning.

What is published: Actions ran about 23 million jobs a day in early 2024; GitHub rebuilt its core backend aiming "to scale 10x over existing usage", and by December 2025 the new architecture handled 71 million a day, with individual enterprises able "to start 7x more jobs per minute"; Octoverse 2026 counts 124 million a day on average over the year to September 2026. GitHub's docs: "each GitHub-hosted runner is a new virtual machine" (single-CPU runners share a VM in containers); eligible Linux jobs may run on AWS or Azure; hosted runners "execute code within ephemeral and clean isolated virtual machines", while self-hosted runners "should almost never be used for public repositories". Limits include 6 hours per job, 256 jobs per matrix, 1,500 triggering events per 10 seconds per repository, and 20 (Free), 60 (Team) or 500 (Enterprise) concurrent standard jobs. Forks get no secrets and a read-only GITHUB_TOKEN. In 2022 GitHub described its run orchestrator as "a durable state machine" that, each time it wakes up, replays execution from the beginning to rebuild its state.

GitHub's availability reports show where it bends. On 2 February 2026, a policy wrongly applied in the compute provider "blocked access to critical VM metadata", so no VM could be created and hosted runners were down for hours in every region, while self-hosted runners elsewhere kept working. On 6 August 2026, a routine deployment briefly cut capacity; during recovery "runners were handed jobs that had already been revoked, then got stuck retrying them... which created a self-amplifying backlog". On 26 August 2026, a burst of events saturated a shared database and "there was no automatic circuit breaker to throttle inbound actions load". GitHub's summary of the 26 August incident: "our shared infrastructure services have not kept up with our month-over-month actions growth and peak load."

%% title: A hosted job's life
stateDiagram-v2
  [*] --> PENDING_APPROVAL: fork PR, first-time contributor
  [*] --> QUEUED: trigger admitted
  PENDING_APPROVAL --> QUEUED: maintainer approves
  PENDING_APPROVAL --> EXPIRED: 30 days
  QUEUED --> ASSIGNED: fair-share pick, lease epoch n
  ASSIGNED --> QUEUED: lease lapsed, epoch n+1
  ASSIGNED --> RUNNING: VM accepts, epoch matches
  RUNNING --> COMPLETED: steps done
  RUNNING --> FAILED: step failed or 6 h limit
  QUEUED --> CANCELLED: user or concurrency group
  COMPLETED --> [*]
  FAILED --> [*]
  CANCELLED --> [*]
  EXPIRED --> [*]
  class COMPLETED ok
  class FAILED bad
  class ASSIGNED queue

• Admit, then queue per account. Triggers pass per-repository and per-account rate limits; admitted jobs wait in the account's queue, and its concurrency cap is its share of the fleet.
• Schedule across accounts. Take the next job from the account furthest below its share (deficit round robin), per runner label. One enterprise's burst raises its own queue time, not everyone's.
• Warm pools per label. Keep booted, never-used VMs sized from a 15-minute forecast per image and size, top up reactively, and destroy each VM after its job. The pool is the price of isolation; size it with Week 10's capacity planning.
• Leases with epochs. An assignment carries an epoch the VM must present to start and report. A lapsed lease returns the job to the queue with a new epoch; a runner holding the old one is refused at once. That turns the August 2026 pattern into one fast rejection.
• A breaker on intake. When the job database's latency or lag crosses a threshold, slow intake automatically and let queues absorb the burst; delayed jobs beat a saturated primary.
• Cells and narrow tokens. Split the control plane into cells of accounts with their own databases and queues; give each job a token scoped to its repository that expires with the job.

# actions/scheduler.py

"""
Fair-share job assignment with leases (illustrative): the next job comes from the
account furthest below its share, and a lease carries an epoch so a runner holding
a revoked assignment is refused at once.
"""

import time

LEASE_S = 120


def pick_next(accounts: dict, label: str):
    ready = [a for a in accounts.values()
             if a.queue(label) and a.running < a.max_concurrent]
    if not ready:
        return None
    a = min(ready, key=lambda a: a.used_recently / a.weight)     # deficit-style fairness
    return a.queue(label).pop()


def assign(job, vm, store) -> dict:
    epoch = store.incr(f"job:{job.id}:epoch")
    store.put(f"lease:{job.id}", {"vm": vm.id, "epoch": epoch, "until": time.time() + LEASE_S})
    return {"job": job.id, "epoch": epoch, "token": mint_token(job.repo, job.permissions, ttl_s=job.max_s)}


def runner_report(store, job_id: str, epoch: int, vm_id: str, status: str) -> str:
    lease = store.get(f"lease:{job_id}")
    if lease is None or lease["epoch"] != epoch or lease["vm"] != vm_id:
        return "REVOKED"                                # stop now, take new work
    store.put(f"job:{job_id}:status", status)
    return "OK"

Deep Dive 6: Searching All the Code (Week 7 Concepts: Search Fundamentals, Indexing Pipeline, Search Operations)#

This builds on Week 7, Day 1 · Search Fundamentals and Architecture, Week 7, Day 2 · Indexing Pipeline and Week 7, Day 5 · Search Operations and Scale.

What is published: GitHub built Blackbird "from scratch, in Rust" because general text search gave a poor experience for code, indexed slowly and cost too much; its first Elasticsearch index of about 8 million repositories "took months". At the 2023 beta it covered "almost 45 million repositories, representing 115 TB of code and 15.5 billion documents". It is sharded by Git blob id, delta-encodes similar repositories so only unique blobs are crawled, and uses Kafka so that "each shard consumes a single Kafka partition", giving results "consistent on a commit-level basis". It indexes variable-length "sparse grams" rather than plain trigrams. Queries are rewritten with permission clauses, fanned out to every shard, re-sorted and permission-checked again. Per-shard p99 is on the order of 100 ms (an upper bound of about 640 queries a second per 64-core host); ingest runs at about 120,000 documents a second, re-indexing everything in about 18 hours; 115 TB became 28 TB of unique content and a 25 TB index. On 1 April 2026 code search was fully down for 2 hours 20 minutes; GitHub reset the index to an earlier point and re-indexed, since "the search index is a secondary index derived from Git repositories".

%% title: Indexing code: from a push to every shard
flowchart TD
  EV@{ icon: "tech:kafka", label: "Push events<br/>repo, new commit" }
  CR["Crawlers<br/>diff against delta parent, new blobs only"]:::service
  SYM["Symbol extraction<br/>definitions, references"]:::service
  DOC@{ icon: "tech:kafka", label: "Documents topic<br/>partitioned by blob id" }
  SH["Index shards<br/>one partition each, sparse grams"]:::core
  CMP@{ icon: "sd:database", label: "Compaction<br/>merge segments, rank order" }
  Q["Query service<br/>parse, add permissions, fan out"]:::hot
  EV ==> CR ==> SYM ==> DOC ==> SH ==> CMP
  Q ==>|"every shard"| SH
  class EV,DOC queue
  class CMP store

• Content-addressed sharding means a fork adds only metadata (one more location for the same blob), not another document. The price: every query touches every shard, so tail latency comes from the slowest one; replicate shards and hedge slow requests.
• Consistency by commit. Because each shard consumes its Kafka partition in order, a repository's results reflect one commit or the previous one, never half of each.
• Permissions twice: in the query, so shards skip invisible documents, and on the results, so a stale permission cache can't leak a private file.
• A secondary index must be cheap to rebuild: re-index from Git at a known rate and keep point-in-time rollbacks.
• Retries need a budget, in any search system. On 12 August 2025 "GitHub search" (the report describes broken issue, pull request and project pages and filters, not code search) was degraded for nearly four hours after retry queues "eventually overwhelmed the load balancers".

# search/query.py

"""
One shard's part of a code search (illustrative): n-gram posting lists are
intersected, locations are filtered to what the searcher may see, and candidates
are verified against the content.
"""

def grams(s: str, n: int = 3) -> set:
    return {s[i:i + n] for i in range(len(s) - n + 1)}


def search_shard(shard, needle: str, visible_repos: set, limit: int = 100) -> list:
    postings = [shard.postings(g) for g in grams(needle)]          # sorted doc ids, best first
    candidates = set.intersection(*map(set, postings)) if postings else set()
    hits = []
    for doc in sorted(candidates):                                  # ids are assigned by rank
        locations = [loc for loc in shard.locations(doc)            # (repo, path, ref) for this blob
                     if loc.public or loc.repo_id in visible_repos]
        if locations and needle in shard.content(doc):              # drop trigram false positives
            hits.append((doc, locations))
            if len(hits) == limit:
                break
    return hits

Nandini's Release Day#

Edge Cases#

Monitoring Dashboard#

SLOs for the Platform#

The interviewer leans back. "One more."

Summary: Concepts Applied from 10-Week Course#

Why GitHub Matters#

Self-Assessment Checklist#

After studying this case study, you should be able to:

Storage and Git traffic:

Events, pull requests and webhooks:

Actions and search:

Sources#

GitHub Engineering: Git storage and traffic:

• Blog

  Patrick Reynolds — Introducing DGit, GitHub Blog, 5 April 2016: https://github.blog/engineering/infrastructure/introducing-dgit/

• Blog

  Patrick Reynolds — Building resilience in Spokes, GitHub Blog, 7 September 2016: https://github.blog/engineering/infrastructure/building-resilience-in-spokes/

• Blog

  Michael Haggerty — Stretching Spokes, GitHub Blog, 13 October 2017: https://github.blog/engineering/infrastructure/stretching-spokes/

• Blog

  Scott Arbeit — Improving large monorepo performance on GitHub, GitHub Blog, 16 March 2021: https://github.blog/engineering/infrastructure/improving-large-monorepo-performance-on-github/

• Blog

  Taylor Blau — Scaling monorepo maintenance, GitHub Blog, 29 April 2021: https://github.blog/engineering/infrastructure/scaling-monorepo-maintenance/

• Blog

  Derrick Stolee — Get up to speed with partial clone and shallow clone, GitHub Blog, 21 December 2020: https://github.blog/open-source/git/get-up-to-speed-with-partial-clone-and-shallow-clone/

• Blog

  Theo Julienne — GLB: GitHub's open source load balancer, GitHub Blog, 8 August 2018: https://github.blog/engineering/infrastructure/glb-director-open-source-load-balancer/

• Blog

  Brian Celenza — Building Git infrastructure for agent-scale development, GitHub Blog, 6 October 2026: https://github.blog/engineering/architecture-optimization/building-git-infrastructure-for-agent-scale-development/

• Blog

  Eleanor White and Victoria Dye — The bridge to agent-scale Git infrastructure, GitHub Blog, 21 October 2026: https://github.blog/engineering/architecture-optimization/the-bridge-to-agent-scale-git-infrastructure/

• Blog

  Mike Hanley — We updated our RSA SSH host key, GitHub Blog, March 2023: https://github.blog/news-insights/company-news/we-updated-our-rsa-ssh-host-key/

GitHub Engineering: events, pull requests, metadata:

• Blog

  Will Haltom — How we improved push processing on GitHub, GitHub Blog, 11 June 2024: https://github.blog/engineering/architecture-optimization/how-we-improved-push-processing-on-github/

• Blog

  GitHub — How GitHub uses merge queue to ship hundreds of changes every day, GitHub Blog, 6 March 2024: https://github.blog/engineering/engineering-principles/how-github-uses-merge-queue-to-ship-hundreds-of-changes-every-day/

• Blog

  Thomas Maurer — Partitioning GitHub's relational databases to handle scale, GitHub Blog, 27 September 2021: https://github.blog/engineering/infrastructure/partitioning-githubs-relational-databases-scale/

• Blog

  GitHub — Upgrading GitHub.com to MySQL 8.0, GitHub Blog, 7 December 2023: https://github.blog/engineering/infrastructure/upgrading-github-com-to-mysql-8-0/

• Blog

  Jason Warner — October 21 post-incident analysis, GitHub Blog, October 2018: https://github.blog/news-insights/company-news/oct21-post-incident-analysis/

GitHub Actions:

• Talk

  Ben De St Paer-Gotch — Let's talk about GitHub Actions, GitHub Blog, 11 December 2025: https://github.blog/news-insights/product-news/lets-talk-about-github-actions/

• Blog

  Aiqiao Yan — How we tripled max concurrent jobs to boost performance of GitHub Actions, GitHub Blog, 16 September 2022: https://github.blog/engineering/architecture-optimization/how-we-tripled-max-concurrent-jobs-to-boost-performance-of-github-actions/

• Docs

  GitHub Docs — GitHub-hosted runners: https://docs.github.com/en/actions/concepts/runners/github-hosted-runners

• Docs

  GitHub Docs — Secure use reference: https://docs.github.com/en/actions/reference/security/secure-use

• Docs

  GitHub Docs — Actions limits: https://docs.github.com/en/actions/reference/limits

• Docs

  GitHub Docs — Events that trigger workflows: https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows

• Docs

  GitHub Docs — Approving workflow runs from forks: https://docs.github.com/en/actions/how-tos/manage-workflow-runs/approve-runs-from-forks

• Docs

  GitHub Docs — Actions runner pricing: https://docs.github.com/en/billing/reference/actions-runner-pricing

Webhooks and merge queue:

• Docs

  GitHub Docs — Best practices for using webhooks: https://docs.github.com/en/webhooks/using-webhooks/best-practices-for-using-webhooks

• Docs

  GitHub Docs — Handling failed webhook deliveries and Redelivering webhooks: https://docs.github.com/en/webhooks/using-webhooks/handling-failed-webhook-deliveries

• Docs

  GitHub Docs — Webhook events and payloads: https://docs.github.com/en/webhooks/webhook-events-and-payloads

• Docs

  GitHub Docs — Managing a merge queue: https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue

Code search:

• Blog

  Timothy Clem — The technology behind GitHub's new code search, GitHub Blog, 6 February 2023: https://github.blog/engineering/architecture-optimization/the-technology-behind-githubs-new-code-search/

Octoverse and availability reports:

• Blog

  GitHub — Octoverse: Commits, pushes, reach new highs in 2026 as developers build with agents, 28 October 2026: https://github.blog/news-insights/octoverse/octoverse-commits-pushes-reach-new-highs-in-2026-as-developers-build-with-agents/

• Blog

  GitHub — Octoverse 2025: A new developer joins GitHub every second, 28 October 2025: https://github.blog/news-insights/octoverse/octoverse-a-new-developer-joins-github-every-second-as-ai-leads-typescript-to-1/

• Blog

  Jakub Oleksy — GitHub availability report: September 2026, 14 October 2026: https://github.blog/news-insights/company-news/github-availability-report-september-2026/

• Blog

  Jakub Oleksy — GitHub availability report: August 2026, 9 September 2026: https://github.blog/news-insights/company-news/github-availability-report-august-2026/

• Blog

  GitHub availability report: April 2026, 14 May 2026: https://github.blog/news-insights/company-news/github-availability-report-april-2026/

• Blog

  GitHub availability report: February 2026, 11 March 2026: https://github.blog/news-insights/company-news/github-availability-report-february-2026/

• Blog

  GitHub availability report: August 2025, 11 September 2025: https://github.blog/news-insights/company-news/github-availability-report-august-2025/

Design note: the architecture, code, thresholds, dashboards and every number labelled as an assumption or estimate are an illustrative design for the problem, not a description of GitHub's internal systems beyond what GitHub has published. The peak multipliers, average repository size, average job length, VM boot time, webhook event and delivery counts, the scheduler, the pack cache, the webhook queueing design and the final challenge's figures are assumptions. Nandini, her library and her co-maintainer are fictional.

Further Reading#

Related systems to study:

• Book

  AWS S3 (Bonus Problem 8 in this series): durability by copies and erasure coding, for objects that never change

• Read

  Google Search (Bonus Problem 5): an inverted index over the web, sharded and served in milliseconds

• Read

  Razorpay (Bonus Problem 18): webhooks to merchants, signatures and idempotent receivers from the payments side

• Read

  ChatGPT (Bonus Problem 14): another fleet of expensive machines shared fairly between tenants

War story:

• Read

  A 43-second network blip gave GitHub 24 hours of trouble: cross-region failover of MySQL primaries, and why webhooks were paused

Foundational concepts (this course):

• Read

  Week 1, Day 2 — Replication Trade-offs

• Read

  Week 1, Day 4 — Hot Keys and Skew

• Read

  Week 2, Day 2 — Idempotency in Practice

• Read

  Week 2, Day 3 — Circuit Breakers

• Read

  Week 2, Day 4 — Webhook Delivery System

• Read

  Week 3, Day 1 — Queue vs Stream

• Read

  Week 3, Day 4 — Dead Letters and Poison Pills

• Read

  Week 4, Day 5 — Multi-Tier Caching

• Read

  Week 5, Day 5 — Leader Election

• Read

  Week 7, Day 2 — Indexing Pipeline

• Read

  Week 9, Day 1 — Tenant Isolation

• Read

  Week 9, Day 2 — Noisy Neighbor Prevention

• Read

  Week 10, Day 4 — Capacity Planning

"Agree on the reference, broadcast the event, and run strangers' code on machines you throw away."