AWS S3 design sheet · System Design Mastery

At a glance#

Number	Value	What it drives
Objects	500T+	~500 KB average → ~250 EB of data
Durability	11 nines (design target)	~5,000 objects a year at exactly 11 nines: a floor to beat
Requests	200M+ / s	~17 trillion a day
Per prefix	3,500 PUT · 5,500 GET / s	partitions split on load; spread hot work over prefixes
Overhead	1.6× with 10 + 6 erasure coding	~400 EB raw vs ~750 EB with 3 copies
Metadata	~1 KB / object · ~500 PB	its own partitioned index layer
Availability	99.99 % designed, 3+ AZs	first byte 100–200 ms for small objects

Architecture#

DiagramS3 on one page: index the keys, erasure-code the bytes

%% title: S3 on one page: index the keys, erasure-code the bytes
flowchart TD
  CL@{ icon: "sd:code", label: "Clients<br/>SDK · CLI · REST · console" } --> LB@{ icon: "sd:load-balancer", label: "DNS → load balancers<br/>regional endpoint" }
  LB --> FE["Front end<br/>SigV4 auth · IAM + bucket policy<br/>request router"]:::service
  FE ==> IX["Index layer · partition engine<br/>hash bucket, range by key prefix<br/>split hot partitions"]:::hot
  IX --> MD@{ icon: "sd:database", label: "Metadata store<br/>key → version → 16 shard locations<br/>quorum writes · witness" }
  IX ==> ST["Storage layer<br/>Reed-Solomon 10 + 6 · checksums<br/>bracketing before ack"]:::core
  IX --> LC["Lifecycle manager<br/>versions · retention · tiering"]:::service
  ST ==> A1@{ icon: "sd:servers", label: "AZ-1<br/>≤ 6 shards" }
  ST ==> A2@{ icon: "sd:servers", label: "AZ-2<br/>≤ 6 shards" }
  ST ==> A3@{ icon: "sd:servers", label: "AZ-3<br/>≤ 6 shards" }
  A1 & A2 & A3 -.->|"lost shard"| RP["Repair service<br/>rebuild in parallel on spare space"]:::service
  LC --> AR@{ icon: "sd:object-storage", label: "Colder classes<br/>Standard-IA · Glacier tiers" }
  class CL client
  class LB edge
  class MD,A1,A2,A3,AR store

Index and bytes scale apart: small, hot, partitioned records on one side; huge, mostly idle bytes on tens of millions of drives on the other.
A PUT succeeds only after the shards are verified and the metadata is committed; that is what makes read-after-write possible.

DiagramErasure coding: any 10 of 16 shards rebuild the object

%% title: Erasure coding: any 10 of 16 shards rebuild the object
flowchart TD
  OB@{ icon: "sd:file", label: "1 MB object" } --> EN["Reed-Solomon 10 + 6<br/>10 data + 6 parity · ~100 KB each"]:::core
  EN --> Z1@{ icon: "sd:servers", label: "AZ-1 · 6 shards" }
  EN --> Z2@{ icon: "sd:servers", label: "AZ-2 · 5 shards" }
  EN --> Z3@{ icon: "sd:servers", label: "AZ-3 · 5 shards" }
  Z1 & Z2 & Z3 --> RD{"Shards that pass checksum?"}
  RD -->|"16 · healthy"| OK["Serve"]:::ok
  RD -->|"10–15 · degraded"| DG["Decode and serve<br/>repair queue rebuilds the rest"]:::queue
  RD -->|"< 10"| BAD["Unreadable<br/>e.g. AZ down + 1 more drive"]:::bad
  class OB client
  class Z1,Z2,Z3 store

Key flows#

DiagramPUT: verify the shards, commit the index, then 200 OK

%% title: PUT: verify the shards, commit the index, then 200 OK
sequenceDiagram
  autonumber
  participant C as Client
  participant F as Front end
  participant I as Index layer
  participant S as Storage layer
  participant M as Metadata replicas
  C->>F: PUT /bucket/key · SHA-256 checksum
  Note over F: SigV4 · bucket policy
  F->>I: Route by bucket + key prefix
  Note over I: lock key · pick 16 locations, ≤ 6 per AZ
  I->>S: Write object
  Note over S: 10 + 6 shards across 3 AZs · checksum each
  Note over S: bracketing: decode back, compare checksum
  S-->>I: Shards durable
  I->>M: Commit key → new version (sync)
  M-->>I: Majority ack
  I-->>F: Committed
  F-->>C: 200 OK
  Note over C,M: any later read sees this version

DiagramGET right after an overwrite: the witness catches stale cache

%% title: GET right after an overwrite: the witness catches stale cache
sequenceDiagram
  autonumber
  participant C as Client
  participant F as Front-end node
  participant W as Witness
  participant M as Metadata replicas
  participant S as Storage layer
  C->>F: GET /bucket/key
  F->>W: Is my cached version the latest?
  alt cache is current
    W-->>F: yes
  else overwritten since cached
    W-->>F: no · newer write exists
    F->>M: Read majority, keep newest version
    M-->>F: v2 · shard locations
  end
  F->>S: Fetch any 10 of 16 shards
  S-->>F: Decoded bytes
  F-->>C: 200 OK · v2, never stale

DiagramHot prefix: 503, back off, partition splits

%% title: Hot prefix: 503, back off, partition splits
sequenceDiagram
  autonumber
  participant J as Batch job
  participant F as Front end
  participant P as Index partition logs/
  J->>F: 20,000 PUT / s, all under logs/
  F->>P: Writes
  Note over P: above 3,500 PUT / s per prefix
  P-->>J: 503 Slow Down (some requests)
  Note over P: request rate over threshold · split by prefix range
  alt client backs off
    J->>F: Retry after 100 ms, then 200, 400… max 5
    F-->>J: 200 OK as partitions split
  else client retries in a tight loop
    Note over J,P: more load on the hot partition · more 503s
  end
  Note over J: lasting fix: logs/part-00/ … part-15/ in parallel

Data model#

DiagramCore entities

%% title: Core entities
classDiagram
  class Bucket {
    +name : string · hashed to a partition
    +region : string
    +versioning : on / off
    +objectLock : retention · compliance mode
  }
  class ObjectVersion {
    +key : string
    +versionId : monotonic · sorts in write order
    +etag : string
    +size : bytes
    +storageClass : string
    +lastModified : timestamp
    +deleteMarker : bool
  }
  class Shard {
    +index : 0 to 15 · 10 data + 6 parity
    +node : string
    +az : string
    +checksum : SHA-256
  }
  class IndexPartition {
    +partitionId : string
    +prefixRange : start to end
    +requestRate : per s · split near 5,500 GET
    +node : string
  }
  class LifecycleRule {
    +prefixFilter : string
    +transitions : days → class
    +expirationDays : int
  }
  Bucket "1" --> "many" ObjectVersion : holds
  ObjectVersion "1" --> "16" Shard : stored as
  IndexPartition "1" --> "many" ObjectVersion : indexes, sorted by key
  Bucket "1" --> "many" LifecycleRule : applies

Partition key: hash of the bucket, then key-prefix ranges sorted inside, so a LIST is a scan of matching keys, not a scatter-gather.
~1 KB of metadata an object (~500 PB total) vs ~500 KB of data; version IDs must sort in write order so "newest" is well defined.

Lifecycle#

DiagramObject version state machine

%% title: Object version state machine
stateDiagram-v2
  [*] --> UPLOADING: PUT or multipart
  UPLOADING --> FAILED: bracketing or quorum failed
  UPLOADING --> CURRENT: shards verified · index committed
  CURRENT --> DEGRADED: shard lost
  DEGRADED --> CURRENT: repair rebuilt it
  CURRENT --> ARCHIVED: lifecycle → IA / Glacier
  CURRENT --> NONCURRENT: overwritten, versioning on
  CURRENT --> DELETE_MARKED: plain DELETE
  DELETE_MARKED --> CURRENT: marker removed
  NONCURRENT --> EXPIRED: lifecycle expiry
  ARCHIVED --> EXPIRED: retention ended
  FAILED --> [*]
  EXPIRED --> [*]
  class CURRENT ok
  class FAILED,EXPIRED bad
  class DEGRADED,DELETE_MARKED queue

Decisions#

Decision	Chose	Over	Because
Redundancy	Erasure coding 10 + 6	3 full copies	1.6× vs 3×: ~400 EB vs ~750 EB
Placement	≤ 6 shards per AZ	Fill by free space	a whole-AZ outage stays inside the parity budget
Write ack	After bracketing + metadata commit	On receipt	never 200 for bytes you can't decode or find
Consistency	Sync metadata + witness check per read	Async metadata	read-after-write with the same throughput
Index	Hash bucket, range by prefix, split on load	Hash every key	prefix LIST stays a sorted scan
Same-key writes	Last-writer-wins + If-Match	Lock every key	cheap default, optimistic locking when needed
Cost	Storage classes + lifecycle rules	One tier	Deep Archive is ~23× cheaper per GB than Standard

When it breaks#

Failure	What users see	Contain it with
A whole AZ loses power	brief availability blip, no loss	≤ 6 shards per AZ, repair starts at once
Hundreds of drives fail daily, or a heat wave	nothing	parallel repair across thousands of drives, spare space
Hot prefix	503 Slow Down	backoff from 100 ms, auto split, spread prefixes
One object gets millions of requests	throttling	front-end cache, more replicas, CDN in front
Stolen keys delete everything	data gone; durability can't help	versioning, MFA Delete, Object Lock, copy in another account
Repair queue grows	nothing yet	critical alert at 1M objects; under-protected while waiting

Say it in the interview#

"Separate the index from the bytes." Small partitioned metadata scales one way, exabytes of shards another.
"11 nines is a floor here." 500T × 10⁻¹¹ is ~5,000 lost objects a year, so the design must beat it.
Erasure coding beats copies: 10 + 6 is 1.6× and, with ≤ 6 shards per AZ, survives a whole AZ.
Repair speed is durability: with hundreds of drives failing a day, what matters is how fast lost shards are rebuilt.
Don't ack what you can't read back: bracketing decodes the shards, then the index commits, then 200 OK.
Strong consistency is a cheap check: every read asks the witness if its cached metadata is current.
Durability isn't protection: a signed DELETE is valid; versioning, MFA Delete and Object Lock save the customer.

AWS S3, the whole design.