Paste some text, get a short link, share it. Pastebin looks like a weekend project, which is why interviewers like it: the interesting parts hide in the details. How do you mint IDs nobody can guess? Where do gigabytes of text live? What happens to a paste that should vanish in ten minutes, or one that goes viral?
The Pastebin interview end to end: immutable pastes with optional expiry and unlisted links; about 12 writes and 116 reads a second but 10 GB of content a day, ~18 TB in five years, so metadata goes in PostgreSQL and content in S3, with reads served by the CDN. Deep dives compare counter and random IDs (8 base62 characters, 2.2 × 10^14 of them, retry on the rare collision) and make expiry exact with a check on every read, cache TTLs capped at expiry and a background sweeper.
Phase 1: Requirements (5 minutes)#
"Can a paste be edited after it's created?"
"No. Create, read, and optionally delete or expire."
"That's a gift: immutable content can be cached forever at the edge. And unlisted pastes: is the link itself the only protection?"
"Yes. Anyone with the link can read an unlisted paste, nobody without it should be able to find it."
- Create a paste (text, up to 10 MB) and get back a short URL
- Read a paste by its URL, rendered or as raw text
- Optional expiry: 10 minutes, 1 day, 1 month, never
- Visibility: public, unlisted (link only) or private (owner only)
- The owner can delete their paste
- Out of scope: editing, full-text search, comments
- Durability: an acknowledged paste is never lost before it expires
- Read latency: p99 under 200 ms for a read, anywhere in the world
- Unguessable IDs: you can't walk the ID space to find unlisted pastes
- Availability: reads matter most; writes can degrade briefly
- Abuse: rate limits and size caps; no one fills the disk for free
Phase 2: Estimation (5 minutes)#
- New pastes: 1M/day ÷ 86,400 s ≈ 12 writes/s, peak ~5x ≈ 60 writes/s
- Reads at 10 : 1: 10M/day ≈ 116 reads/s, peak ≈ 600 reads/s
- Average paste: 10 KB (most are small; the 10 MB cap is rare)
- Content per day: 1M × 10 KB = 10 GB/day → ~3.6 TB/year → ~18 TB in 5 years
- Metadata per paste (ID, owner, expiry, visibility, size, pointer): ~200 bytes → 200 MB/day → ~365 GB in 5 years
- Pastes in 5 years: ~1.8 billion
- ID space with 8 base62 characters: 62^8 ≈ 2.2 × 10^14, about 120,000 times the number of pastes we'll ever store
"The traffic is tiny; one server could take it. The interesting numbers are storage (terabytes of blobs, which don't belong in a database) and the ID space (big enough that random IDs are both safe and unguessable)."
Phase 3: API (3 minutes)#
POST /v1/pastes
{ "content": "...", "expires_in": "1d", "visibility": "unlisted", "language": "python" }
→ 201 { "id": "aZ3k9QxP", "url": "https://paste.example/aZ3k9QxP", "delete_token": "..." }
GET /v1/pastes/aZ3k9QxP → 200 { metadata, content } (or 404 if missing / expired)
GET /raw/aZ3k9QxP → 200 text/plain (served by the CDN)
DELETE /v1/pastes/aZ3k9QxP → 204 (owner session or delete_token)Anonymous users get a delete_token once, at creation. It's the only way to delete their paste later, so store only its hash.
Phase 4: High-level design (10 minutes)#
%% title: Pastebin: metadata in a database, content in object storage, reads at the edge
flowchart TD
U@{ icon: "sd:browser", label: "Browser / curl" }
CDN@{ icon: "sd:cdn", label: "CDN<br/>caches immutable pastes" }
API["Paste service<br/>create · read · delete"]:::core
R@{ icon: "tech:redis", label: "Redis<br/>hot metadata" }
PG@{ icon: "tech:postgresql", label: "PostgreSQL<br/>paste metadata" }
S3@{ icon: "tech:s3", label: "S3<br/>paste content" }
SW@{ icon: "sd:cron", label: "Expiry sweeper<br/>every minute" }
U --> CDN
CDN -->|"miss"| API
API --> R
API --> PG
API -->|"put / get blob"| S3
SW --> PG
SW -->|"delete + purge"| S3
class U client
class CDN edge
class R cache
class PG,S3 store
class SW service- Create. The service validates size and rate limits, generates a random 8-character ID, and uploads the content to object storage under
pastes/aZ3k9QxP. - Commit. It inserts the metadata row (
id, owner, visibility, expires_at, size, blob_key) only after the upload succeeds, so a row never points at a missing blob. An orphaned blob from a failed insert is cleaned up later. - Read at the edge.
GET /raw/aZ3k9QxPhits the CDN first. Content never changes, so the edge can keep it until the paste expires. - Read on a miss. The service checks Redis, then PostgreSQL, for the metadata: does it exist, has it expired, may this user see it? Then it streams the blob from object storage.
Why split metadata and content? The database holds 1.8 billion small rows that need indexes (by owner, by expiry). The object store holds 18 TB of opaque bytes it can keep cheaply and durably. Putting 10 MB text blobs into database rows bloats backups, replication and the buffer cache for no benefit. Small pastes (say under 4 KB) can be stored inline in the row to save a round trip. That's an optimisation, not the core design.
Phase 5: Deep dives (15 minutes)#
Deep dive 1: Generating IDs#
Counter, hash or random?
Three ways to mint an ID: an auto-increment counter encoded in base62, a hash of the content, or 8 random base62 characters. Which one breaks the "unlisted" requirement, which one surprises users, and what does the third cost?
One strong answer
- Counter + base62: short and collision-free, but sequential. Anyone can try
aZ3k9QxP,aZ3k9QxQ, … and read every unlisted paste. It breaks the core privacy promise. - Hash of content: the same text always gets the same ID, which dedupes, but it also reveals that someone else pasted exactly this text, and two people's different expiry settings collide on one row.
- Random 8 characters: unguessable, independent of content, and no coordination between servers. The cost is a possible collision: with 1.8 billion IDs in a space of 2.2 × 10^14, a new ID collides about 1 time in 120,000. Insert with "fail if exists" and retry with a new ID.
So the write path is:
INSERT INTO pastes (id, owner_id, visibility, expires_at, size_bytes, blob_key)
VALUES ($random_id, $owner, 'unlisted', now() + interval '1 day', 10240, 'pastes/' || $random_id)
ON CONFLICT (id) DO NOTHING
RETURNING id; -- no row returned → collision → generate a new ID and retryThe random part must come from a cryptographically secure generator. A predictable PRNG seeded with the time makes IDs guessable again.
An alternative you should mention: a key generation service that pre-generates unused random IDs into a table and hands them out in batches. It removes the retry but adds a service to run and keep available. At 60 writes a second, the retry is cheaper.
Deep dive 2: Expiry, deletion and the cache#
Expiry looks trivial ("delete it after a day") but touches every layer: the database row, the blob, Redis and every CDN edge.
1. Check on read (correctness)
Every read compares expires_at with the current time and returns 404 if it has passed, even if the row still exists. This alone makes expiry exact, whatever the cleanup lag.
2. Cap every cache TTL at the expiry
The CDN and Redis TTL for a paste is min(default TTL, expires_at − now). A paste expiring in 10 minutes is never cached for longer than 10 minutes.
3. Sweep in the background (space)
A sweeper runs every minute: select a batch of IDs with expires_at < now() using an index on expires_at, delete the blobs, then the rows. Small batches keep it from locking the table. Object-storage lifecycle rules can act as a safety net.
4. Explicit deletes purge the edge
An owner's delete removes the row and blob and sends a CDN purge for that URL. Until the purge lands (often seconds), the edge may still serve it. If that's unacceptable for private pastes, don't cache private pastes at the CDN at all.
Only one instance of the sweeper should run at a time (or each instance takes a disjoint range of IDs), and it must be safe to run twice: deleting an already-deleted blob is a no-op. That's the distributed cron problem in miniature.
Burn after reading (a paste that self-destructs on its first view) breaks the CDN design: it can't be cached at all, and the first read must atomically mark it consumed (UPDATE … SET burned = true WHERE id = $1 AND burned = false RETURNING …) so two simultaneous readers can't both see it.
Phase 6: Trade-offs and failure modes (5 minutes)#
A paste goes viral
Millions of reads of one ID: the CDN absorbs them, because content is immutable. Without a CDN, Redis would see a hot key; add a short local cache in the service.
Abuse
Spam, malware hosting and leaked credentials. Per-IP and per-account rate limits on create, a size cap, scanning new pastes asynchronously, and a takedown path that deletes and purges.
Upload succeeds, row insert fails
The blob is orphaned but harmless (no row points at it). A weekly job lists blobs with no row and deletes them.
Metadata database down
Reads of cached pastes keep working from the CDN. New pastes fail; queueing them is possible but returns a link before durability, so it's better to fail fast and retry.
Region loss
Object storage is replicated across zones by the provider; PostgreSQL has a standby in another zone. Cross-region replication is only worth it if the requirements ask for it.
| Decision | We chose | We gave up |
|---|---|---|
| IDs | 8 random base62 chars + retry | Shortest possible links |
| Storage | Metadata in PostgreSQL, content in S3 | One-system simplicity |
| Reads | CDN with TTL capped at expiry | Instant deletion at the edge |
| Expiry | Check on read + background sweep | Space reclaimed exactly on time |
Go deeper in the course#
- Week 1 capstone: URL shortener at scale: Pastebin's closest cousin, ID generation and redirects at far higher traffic
- Week 1, Day 1: Partitioning deep-dive: how the metadata table shards by ID when one database isn't enough
- Week 1, Day 3: Rate limiting at scale: per-IP and per-account limits on create
- Week 4, Day 1: Caching patterns: cache-aside for metadata
- Week 4, Day 5: Multi-tier caching: the CDN, Cache-Control headers and purges
- Week 2, Day 5: Distributed cron: running the expiry sweeper exactly once
- Week 9, Day 4: Right to deletion: deleting data everywhere it was copied, caches included
- Week 0, Part 3: Back-of-the-envelope estimation: the storage and ID-space arithmetic
- Side quest: AWS S3: what's behind the object store holding the content
Check yourself#
Pastebin
Q1/3 Why is an auto-increment counter encoded in base62 a poor choice of paste ID here?
Answer & explanation
Answer: B
A counter is collision-free, but anyone can try neighbouring IDs and read unlisted pastes. Random IDs from a secure generator fix that.
Q2/3 A paste expires in 10 minutes. What should its CDN cache TTL be?
Answer & explanation
Answer: C
Cap every cache TTL at
expires_at − now. Immutable content can be cached for a long time, but never past its expiry.Q3/3 The sweeper is a few hours behind. How does the design still stop expired pastes being read?
Answer & explanation
Answer: C
The read-time check makes expiry exact; the sweeper only reclaims space, so its lag doesn't matter for correctness.
New lessons by email
New lessons, side quests and case files as they ship, at most once a week. No spam, unsubscribe anytime.
Discussion · …
Public: everyone reading this lesson sees comments. Be kind and keep it on the material.