Instagram Feed and Stories.

How Do You Show Three Billion People the Few Posts They Care About Most, in the Half-Second After They Open the App?

It is 9 pm in Bengaluru. Meera opens Instagram on the bus home. She follows 600 accounts: friends from college, her cousin who just had a baby, a dozen chefs, two football clubs and a few hundred more. Since she last looked, this afternoon, they have posted 340 photos, videos and reels, and 90 of them have stories up.

She will scroll for maybe two minutes. In that time she'll see perhaps twenty posts. Her cousin's baby photo, posted three hours ago, has to be one of the first. The stories tray at the top has to show the people she actually watches, with the first story already loaded when she taps. And every one of those photos has to arrive sharp, on a bus, on a phone signal that comes and goes.

This is the system we'll design today.

The Interview Begins#

You're interviewing at a company building a photo-sharing app. The interviewer hands you a phone with the app open. She scrolls: a friend's photo, a reel, a carousel from a restaurant, a suggested post. At the top, a row of circles with coloured rings.

Your Questions#

Functional Requirements#

Non-Functional Requirements#

Traffic Estimation#

Fan-Out Estimation#

Media Estimation#

Key Metrics Summary#

System Architecture#

%% title: Feed and stories architecture
flowchart TD
  subgraph L1["Layer 1 · Clients"]
    direction LR
    AND@{ icon: "tech:android", label: "Android app" }
    IOS@{ icon: "tech:apple", label: "iOS app" }
    WEBC@{ icon: "sd:browser", label: "Web" }
  end
  subgraph L2["Layer 2 · Edge"]
    direction LR
    CDN@{ icon: "sd:cdn", label: "CDN<br/>images and video segments" }
    API@{ icon: "sd:api", label: "API gateway<br/>feed, tray, post" }
  end
  subgraph L3["Layer 3 · Media"]
    direction LR
    MED["Media service<br/>resize, encode, store once"]:::service
    BLOB@{ icon: "sd:object-storage", label: "Blob storage<br/>hot, then warm tiers" }
  end
  subgraph L4["Layer 4 · Posts and inboxes"]
    direction LR
    POST@{ icon: "tech:postgresql", label: "PostgreSQL · posts<br/>sharded by author" }
    FAN@{ icon: "sd:worker", label: "Fan-out workers<br/>post IDs to followers" }
    INB@{ icon: "tech:cassandra", label: "Cassandra · inboxes<br/>recent post IDs per viewer" }
  end
  subgraph L5["Layer 5 · Ranking"]
    direction LR
    FEED["Feed service<br/>candidates, stages, rules"]:::hot
    TRAY["Stories tray<br/>unseen, ranked, 24 h"]:::core
    FS@{ icon: "sd:ml", label: "Models and features<br/>embeddings, counters" }
  end
  AND ==> API
  IOS ==> API
  WEBC ==> API
  AND ==>|"media"| CDN
  API -->|"upload"| MED
  MED --> BLOB
  CDN --> BLOB
  API -->|"create post"| POST
  POST -.->|"new post"| FAN
  FAN ==> INB
  API ==>|"open feed"| FEED
  API --> TRAY
  FEED --> INB
  FEED --> FS
  TRAY --> FS
  class AND,IOS,WEBC client
  class CDN,API edge
  class BLOB,POST,INB store
  class FAN service
  class FS service

KEY INSIGHTS:

• Bytes and IDs travel separately. The media service and CDN move megabytes; the post, inbox and ranking systems move 64-bit IDs and a few fields
• Write once per post, read cheaply per open: resizing, encoding and fan-out happen once; the feed read is an inbox lookup, a ranking pass and a cache-friendly hydrate
• Ranking is its own tier with its own models and features, because it is where most of the compute per read goes

Data Flow: One Photo, From Posting to the Top of Meera's Feed#

%% title: A baby photo, from the cousin's phone to Meera's feed
sequenceDiagram
  autonumber
  participant C as Cousin's app
  participant M as Media service
  participant P as Posts shard
  participant F as Fan-out workers
  participant I as Meera's inbox
  participant R as Feed service
  participant Me as Meera's app
  C->>M: Upload photo, 3.2 MB
  M->>M: Store original, make 1080, 640, 320, thumbnail
  M-->>C: Media keys
  C->>P: Create post with media keys
  P-->>C: Post ID, time-sortable
  P-)F: New post by an account with 420 followers
  F->>I: Append post ID to 420 inboxes
  Note over Me: three hours later
  Me->>R: Open feed
  R->>I: Recent post IDs, newest first
  R->>R: Drop seen, rank in stages
  R-->>Me: Page 1 · the baby photo is first
  Me->>Me: Fetch the 1080 copy from the CDN

MEANWHILE, IN PARALLEL:

• The stories tray loads in its own request; the first two stories' media is prefetched before Meera taps
• Every like, comment and second of viewing time goes back as events that update the counters and features ranking reads

Deep Dive 1: Media, Stored Once and Served at Every Size (Week 4 Concepts: Multi-Tier Caching; Week 1: Replication)#

This builds on Week 4, Day 5 · Multi-Tier Caching.

Videos cost far more than photos. Meta's 2022 post on Instagram video explains why: each upload gets "minimum functionality" encodings that every client can play (basic adaptive-bitrate ladders, plus progressive files for old app versions) and "advanced encodings" with newer compression. Instagram was spending "more than 80 percent" of its video compute on minimum-functionality encodings while advanced ones "covered only 15 percent of total watch time". Its fix was to stop encoding the basic ABR ladder separately: it repackaged the progressive encodings' frames into an ABR-capable structure, which took 0.36 seconds instead of the 86.17 CPU-seconds a 720p encode needed. That cut the cost of basic ABR encodings by 94% and let advanced-encoding watch time coverage grow by 33%.

%% title: One upload through the media pipeline
flowchart TD
  UP(["Upload<br/>photo or video"]):::client
  ORIG@{ icon: "sd:object-storage", label: "Original<br/>kept, never served" }
  subgraph PH["Photos · at upload"]
    direction LR
    SIZES["1080, 640, 320,<br/>thumbnail"]:::service
  end
  subgraph VI["Videos · by watch time"]
    direction LR
    BASIC["Minimum functionality<br/>encode once, repackage"]:::core
    ADV["Advanced encodings<br/>for videos people watch"]:::queue
  end
  HOT@{ icon: "sd:object-storage", label: "Hot tier<br/>new and popular" }
  WARM@{ icon: "sd:object-storage", label: "Warm tier<br/>older, erasure-coded" }
  EDGE@{ icon: "sd:cdn", label: "CDN edges<br/>most reads end here" }
  UP --> ORIG
  ORIG --> SIZES --> HOT
  ORIG --> BASIC --> HOT
  ORIG -.->|"once it is watched enough"| ADV -.-> HOT
  HOT -.->|"after weeks"| WARM
  HOT --> EDGE
  WARM --> EDGE
  class ORIG,HOT,WARM store
  class EDGE edge

• Hot, then warm. A post gets most of its views in its first days. Facebook's f4 paper (2014) describes moving such "warm" blobs from triple replication to erasure-coded storage, cutting the effective replication factor from 3.6 to 2.8 or 2.1. The same idea applies here: after a few weeks, a photo's bytes move to cheaper storage; its key doesn't change.
• Expired stories go warm immediately, since only their author's archive will open them.
• Deletes are tombstones first: the post is hidden at once; the media keys are deleted from storage and purged from the CDN afterwards.

# media/keys.py

"""
Content-addressed media keys (illustrative). The same bytes always get the
same key, so a re-share is stored once and a cached copy never goes stale:
an edited image is new bytes, so a new key.
"""

import hashlib

PHOTO_SIZES = (1080, 640, 320, 150)


def media_key(original: bytes) -> str:
    return hashlib.sha256(original).hexdigest()[:32]


def derived_url(key: str, width: int, fmt: str = "webp") -> str:
    return f"https://cdn.example.com/m/{key}/{width}.{fmt}"


def pick_width(screen_px: int, fraction_of_screen: float) -> int:
    """The smallest stored size that still looks sharp in its slot."""
    need = screen_px * fraction_of_screen
    fits = [w for w in sorted(PHOTO_SIZES) if w >= need]
    return fits[0] if fits else max(PHOTO_SIZES)

Deep Dive 2: Posts, IDs and Inboxes (Week 1 Concepts: Partitioning, Hot Keys; Week 4: Feed Caching)#

This builds on Week 4, Day 4 · Feed Caching and Week 1, Day 1 · Partitioning Deep-Dive.

# posts/ids.py

"""
Time-sortable 64-bit IDs in the layout Instagram published in 2012
(illustrative Python; Instagram generated them in PL/pgSQL inside each
logical shard): 41 bits of milliseconds since an epoch, 13 bits of logical
shard, 10 bits of a per-shard sequence.
"""

import time

EPOCH_MS = 1314220021721          # the epoch in Instagram's published example (August 2011)
SHARD_BITS, SEQ_BITS = 13, 10


def make_id(shard_id: int, seq: int, now_ms: int = None) -> int:
    now_ms = int(time.time() * 1000) if now_ms is None else now_ms
    return ((now_ms - EPOCH_MS) << (SHARD_BITS + SEQ_BITS)) | (shard_id << SEQ_BITS) | (seq % 1024)


def shard_of(post_id: int) -> int:
    return (post_id >> SEQ_BITS) & ((1 << SHARD_BITS) - 1)


def time_of_ms(post_id: int) -> int:
    return (post_id >> (SHARD_BITS + SEQ_BITS)) + EPOCH_MS


def logical_shard(user_id: int, logical_shards: int = 2000) -> int:
    return user_id % logical_shards     # the post's example: user 31341 -> shard 1341

Fan-out: push for most, pull for the few#

%% title: Hybrid fan-out for a new post
flowchart TD
  NEW(["New post<br/>post ID, author"]):::client
  CHK{"Author's followers<br/>above 1 million?"}:::core
  PUSH@{ icon: "sd:worker", label: "Fan-out workers<br/>batches of 1,000 inboxes" }
  INBOX@{ icon: "tech:cassandra", label: "Cassandra · inboxes<br/>newest 1,500 IDs per viewer" }
  CEL@{ icon: "tech:redis", label: "Redis · recent posts<br/>of huge accounts" }
  READ["Feed read<br/>inbox ∪ huge accounts followed"]:::hot
  NEW --> CHK
  CHK -->|"no"| PUSH --> INBOX
  CHK -->|"yes: skip fan-out"| CEL
  INBOX --> READ
  CEL --> READ
  class INBOX store
  class CEL cache
  class PUSH service

• The inbox is bounded: the newest ~1,500 post IDs per viewer, trimmed on write. Nobody scrolls further back than that through new posts; older ones come from elsewhere if needed.
• Fan-out jobs are idempotent: appending the same post ID to an inbox twice is a no-op, so a retried batch is harmless.
• Inactive followers are skipped: fan-out writes only to viewers active in the last month; a returning viewer's inbox is rebuilt by pulling from the accounts they follow.
• Unfollow and block are filters at read time first (drop IDs from that author), and inbox cleanup later.

# feed/fanout.py

"""
Hybrid fan-out (illustrative). Normal accounts push the new post's ID into
followers' inboxes in batches; huge accounts skip fan-out and are pulled at
read time. Every write is idempotent.
"""

HUGE = 1_000_000
BATCH = 1_000
INBOX_MAX = 1_500


def on_new_post(post: dict, graph, inboxes, celeb_cache, active) -> None:
    if graph.follower_count(post["author"]) >= HUGE:
        celeb_cache.add_recent(post["author"], post["id"])        # readers pull it
        return
    for batch in graph.followers(post["author"], batch=BATCH):
        viewers = [v for v in batch if active.recently(v)]
        inboxes.add_many(viewers, post["id"], keep=INBOX_MAX)     # set semantics: retry-safe


def candidates(viewer: int, graph, inboxes, celeb_cache, limit: int = 1_500) -> list:
    ids = set(inboxes.recent(viewer, limit))
    for author in graph.huge_accounts_followed(viewer):           # usually a handful
        ids.update(celeb_cache.recent(author, limit=50))
    return sorted(ids, reverse=True)                               # IDs sort by time

Deep Dive 3: Ranking the Feed in Stages (Week 7 Concepts: Relevance; Week 8: Event Ingestion)#

This builds on Week 7, Day 3 · Query Processing and Relevance.

What Instagram and Meta have published: Instagram began ranking the feed in March 2016, ordering it by "the likelihood you'll be interested in the content, your relationship with the person posting and the timeliness of the post". In 2023 Mosseri described the signals, in rough order of importance (your activity, information about the post, information about the person who posted, your history of interacting with them), and the five predictions: time spent, comment, like, share and profile tap. Meta's 2023 post on Explore describes the funnel shape: retrieval narrows billions of items, a lightweight first stage scores thousands, a heavy multi-task second stage ranks about a hundred, and a final reranking applies integrity filters and diversity rules. It uses "Two Towers" models whose user embeddings can be cached and whose item embeddings sit in approximate-nearest-neighbour indexes.

%% title: The feed ranking funnel for one page
flowchart TD
  C(["1,500 candidates<br/>inbox ∪ huge accounts"]):::client
  F1["Filters<br/>seen, blocked, muted, deleted"]:::service
  S1@{ icon: "sd:ml", label: "First stage · light model<br/>1,200 → 300" }
  S2@{ icon: "sd:ml", label: "Second stage · multi-task<br/>time, comment, like, share, profile tap" }
  V["Value = weighted sum<br/>of the five predictions"]:::core
  RR["Final rerank<br/>diversity, integrity, freshness"]:::core
  OUT(["Page of 10<br/>logged as impressions"]):::ok
  FE@{ icon: "tech:redis", label: "Redis · features<br/>counts, closeness, recent activity" }
  C --> F1 --> S1 --> S2 --> V --> RR --> OUT
  FE -.-> S1
  FE -.-> S2
  class S1,S2 service
  class FE cache

# feed/rank.py

"""
Staged feed ranking (illustrative). Cheap filters, a light model over all
candidates, a heavy multi-task model over the survivors, one value per post
from the five predictions, then rules. Weights come from experiments.
"""

from typing import Dict, List

WEIGHTS = {"time_spent_s": 0.02, "p_comment": 4.0, "p_like": 1.0, "p_share": 3.0, "p_profile_tap": 1.5}
FIRST_KEEP = 300
PAGE = 10
MAX_RUN_PER_AUTHOR = 2


def value(pred: Dict[str, float]) -> float:
    return sum(WEIGHTS[k] * pred[k] for k in WEIGHTS)


def rank_page(viewer: dict, candidate_ids: List[int], deps) -> List[int]:
    posts = deps.posts.get_many(candidate_ids)                       # cached hydrate
    posts = [p for p in posts
             if p["id"] not in viewer["seen"]
             and p["author"] not in viewer["blocked_or_muted"]
             and not p["deleted"]]

    light = deps.light_model.score(viewer, posts)                    # ~0.01 ms each
    survivors = [p for _, p in sorted(zip(light, posts), key=lambda x: x[0], reverse=True)][:FIRST_KEEP]

    preds = deps.heavy_model.predict(viewer, survivors)              # batched multi-task
    scored = sorted(((value(pr), p) for pr, p in zip(preds, survivors)), key=lambda x: x[0], reverse=True)

    page, run_author, run_len = [], None, 0
    for _, p in scored:
        if deps.integrity.blocked(p):
            continue
        if p["author"] == run_author and run_len >= MAX_RUN_PER_AUTHOR:
            continue                                                 # diversity: no long runs
        run_len = run_len + 1 if p["author"] == run_author else 1
        run_author = p["author"]
        page.append(p["id"])
        if len(page) == PAGE:
            break
    deps.impressions.log(viewer["id"], page)                         # labels come back as events
    return page

Features, freshness and the seen state#

• Features are mostly counters: how often Meera comments on her cousin's posts, how many likes a post has gathered in its first hour, how long she watches videos from this account. They are updated from interaction events (Week 8's event ingestion) and read from a cache at ranking time.
• The seen state is a per-viewer set of post IDs that reached the screen, with a short expiry. Without it, the same top post greets her on every open.
• Freshness is a feature, not a sort: a post's age is an input to the model, so a three-hour-old baby photo can outrank a ten-minute-old ad-like post from a brand.
• Fallbacks: if the heavy model is slow or down, serve the first-stage order; if features are missing, serve newest-first from close accounts. A worse feed beats an empty one.

Deep Dive 4: Stories, the Tray and 24 Hours (Week 9 Concepts: Right to Deletion; Week 4: Caching)#

%% title: Building one stories tray
sequenceDiagram
  autonumber
  participant Me as Meera's app
  participant T as Tray service
  participant G as Follow graph
  participant A as Active-story index
  participant S as Seen state
  Me->>T: Open app
  T->>G: Accounts Meera follows · 600
  T->>A: Which have stories in the last 24 h? · 90
  T->>S: Which of those has Meera seen? · 30 fully
  T->>T: Rank the 60 unseen, seen ones last
  T-->>Me: Tray of 90, first 60 ringed
  Me->>Me: Prefetch first frames of the top 3
  Me->>S: Seen story 8812 (on view)

• Audience is part of the index: a close-friends story is indexed with its audience list, so the tray only shows it to people on the list. The check runs at read time, so removing someone from the list hides future reads at once.
• Seen-marks are the write load: ~350,000 a second at peak. They go to a key-value store as small set additions, batched on the phone and sent every few seconds; losing one means a story looks unseen once more, not a correctness failure.
• The viewers list is the same events seen from the author's side: append (story, viewer, time); count and page through it on demand.

# stories/tray.py

"""
The stories tray (illustrative). An active-story index with a 24-hour expiry
per story, a per-viewer seen set with a 48-hour expiry, and a ranked tray
with unseen accounts first. Nothing is deleted at hour 24: entries expire.
"""

import time

STORY_TTL_S = 24 * 3600
SEEN_TTL_S = 48 * 3600


def post_story(author: int, story_id: int, audience, index, archive) -> None:
    index.add(author, story_id, audience=audience, expires_at=time.time() + STORY_TTL_S)
    archive.add(author, story_id)                     # the author's archive keeps it after expiry


def tray(viewer: int, graph, index, seen, ranker) -> list:
    now = time.time()
    active = index.active_for(graph.following(viewer), now)          # {author: [story ids]}
    visible = {a: [s for s in ids if index.audience_allows(s, viewer)] for a, ids in active.items()}
    visible = {a: ids for a, ids in visible.items() if ids}
    seen_ids = seen.get(viewer)
    unseen = [a for a, ids in visible.items() if ids[-1] not in seen_ids]
    watched = [a for a in visible if a not in unseen]
    return ranker.order(viewer, unseen) + ranker.order(viewer, watched)


def mark_seen(viewer: int, story_ids: list, seen) -> None:
    seen.add_many(viewer, story_ids, ttl=SEEN_TTL_S)  # idempotent set adds, batched by the app

A Post's First Day#

Edge Cases#

Monitoring Dashboard#

SLOs for the Platform#

The interviewer glances at the calendar. "One more."

Summary: Concepts Applied from 10-Week Course#

Why Instagram Matters#

Self-Assessment Checklist#

After studying this case study, you should be able to:

Media:

Feed:

Stories and operations:

Sources#

Instagram and Meta:

• Book

  Mark Zuckerberg — Instagram reaches 3 billion monthly active users, 24 September 2025 (as reported by Exchange4Media): https://www.exchange4media.com/digital-news/instagram-crosses-3bn-monthly-users-strengthens-focus-on-reels-and-dms-147825.html

• Blog

  Instagram — See posts you care about first in your feed (people miss about 70% of their feeds; interest, relationship, timeliness), March 2016: https://about.instagram.com/blog/announcements/see-posts-you-care-about-first-in-your-feed

• Blog

  Adam Mosseri — Instagram Ranking Explained (an algorithm per surface; Feed signals and five predictions; Stories signals), May 2023: https://about.instagram.com/blog/announcements/instagram-ranking-explained

• Blog

  Instagram — Introducing Stories Highlights and Stories Archive (stories save to the archive when they expire), December 2017: https://about.instagram.com/blog/announcements/introducing-stories-highlights-and-stories-archive

• Read

  TheWrap — Instagram Stories just passed 500 million daily users (Zuckerberg on Facebook's Q4 2018 earnings call), January 2019: https://thewrap.com/instagram-stories-just-passed-500-million-daily-users

Engineering posts:

• Blog

  Instagram Engineering — What Powers Instagram: Hundreds of Instances, Dozens of Technologies (PostgreSQL, S3 and CloudFront, Redis for the main feed, fan-out in Gearman, 14 million users), December 2011: https://instagram-engineering.com/what-powers-instagram-hundreds-of-instances-dozens-of-technologies-adf2e22da2ad

• Blog

  Instagram Engineering — Sharding & IDs at Instagram (41/13/10-bit IDs, logical shards as PostgreSQL schemas), December 2012: https://instagram-engineering.com/sharding-ids-at-instagram-1cf5a71e5a5c

• Blog

  Instagram Engineering — Open-sourcing a 10x reduction in Apache Cassandra tail latency (Cassandra since 2012 for Feed and the Direct inbox), March 2018: https://instagram-engineering.com/open-sourcing-a-10x-reduction-in-apache-cassandra-tail-latency-d64f86b43589

• Book

  Ryan Peterman, Haixia Shi — Reducing Instagram's basic video compute time by 94 percent (minimum-functionality vs advanced encodings, 80% / 15%, 86.17 s vs 0.36 s, +33% coverage), November 2022: https://engineering.fb.com/2022/11/04/video-engineering/instagram-video-processing-encoding-reduction/

• Blog

  Vladislav Vorotilov, Ilnur Shugaepov — Scaling the Instagram Explore recommendations system (four stages, Two Towers, cached user embeddings, ANN), August 2023: https://engineering.fb.com/2023/08/09/ml-applications/scaling-instagram-explore-recommendations-system/

Research:

• Paper

  Subramanian Muralidhar et al. — f4: Facebook's Warm BLOB Storage System, OSDI 2014 (warm blobs, effective replication 3.6 → 2.8 or 2.1): https://www.usenix.org/conference/osdi14/technical-sessions/presentation/muralidhar

Design note: the architecture, code, thresholds, rates, sizes and dashboards here are an illustrative design for the problem, not a description of Instagram's internal systems beyond what Instagram and Meta have published.

Further Reading#

Related systems to study:

• Read

  Twitter / X (Bonus Problem 10 in this series): fan-out on write versus read, and the celebrity merge, in depth

• Talk

  YouTube (Bonus Problem 19): an upload pipeline, view counting and two-stage recommendations

• Talk

  Netflix (Bonus Problem 6): delivering video from caches near viewers

• Read

  WhatsApp (Bonus Problem 2): media sharing and delivery at messaging scale

Foundational concepts (this course):

• Read

  Week 4, Day 4 — Feed Caching

• Read

  Week 4, Day 5 — Multi-Tier Caching

• Read

  Week 1, Day 1 — Partitioning Deep-Dive

• Read

  Week 1, Day 4 — Hot Keys and Skew

• Read

  Week 7, Day 3 — Query Processing and Relevance

• Read

  Week 9, Day 4 — Right to Deletion

"Every photo is stored once; every feed is a choice of twenty out of thousands, made again each time you open the app."