Slack.

How Do You Run One Chat App for a Million Separate Companies, and Let Them Talk to Each Other?

It is Monday 5 January 2026 in Bengaluru, the first Monday of the year. At 9:52 am Meera Iyer, who runs payments reliability at Paylane, a 6,000-person fintech, opens her laptop for the first time in two weeks. Slack shows its loading screen a little longer than usual. Paylane's Slack is an Enterprise Grid org with fourteen workspaces (Engineering, Ops, Sales, one per city office), and Meera is in nine of them.

At 9:58 a dashboard alert lands in #inc-payments: the morning settlement file from Paylane's partner bank has not arrived. Meera posts "@here settlement file from Kaveri Bank missing, joining the bank's on-call now", and #inc-payments goes bold in 900 sidebars; every member who is active right now, at a laptop or with Slack open on a phone, gets a notification. She switches to #paylane-kaveribank, a channel shared with the bank's own Slack, where the bank's on-call engineer is already typing. Then she searches "settlement file timeout" and finds the thread from last March where the same thing happened, with the fix pinned at the bottom.

At 10:00:00 thousands of reminders fire across Paylane at once ("Stand-up in 15 minutes"). Nobody notices. The messages, the shared channel, the search and the buzzing phones all just work, across two companies and fourteen workspaces, on a morning when half the world is reconnecting with empty caches.

This is the system we'll design today.

The Interview Begins#

You're interviewing at a company that builds workplace software. The interviewer shares a screenshot: a column of workspace icons on the left, a sidebar with "#inc-payments" in bold and a red "3" next to "#paylane-kaveribank", a small two-building icon beside that channel's name, a message with "@here" highlighted, and a search box at the top reading "settlement file timeout".

Your Questions#

Functional Requirements#

Non-Functional Requirements#

Connections and Fan-Out#

Channel Servers#

Storage, Jobs and Search#

Key Metrics Summary#

System Architecture#

%% title: Slack's four paths: write, real time, edge cache, async
flowchart TD
  subgraph L1["Layer 1 · Clients"]
    direction LR
    APP@{ icon: "sd:laptop", label: "Slack clients<br/>desktop, mobile, web" }
  end
  subgraph L2["Layer 2 · Edge regions"]
    direction LR
    ENV@{ icon: "sd:load-balancer", label: "Envoy<br/>TLS, websockets, AZ weights" }
    FL["Flannel<br/>edge cache, lazy boot"]:::service
    GS["Gateway servers<br/>websockets, subscriptions"]:::core
  end
  subgraph L3["Layer 3 · Core region"]
    direction LR
    WEB@{ icon: "sd:api", label: "Webapp · Hack<br/>every API and write" }
    AS["Admin servers<br/>stateless routers"]:::service
    CS["Channel servers<br/>consistent hash, in-memory"]:::hot
    PS["Presence servers<br/>users hashed"]:::service
  end
  subgraph L4["Layer 4 · State"]
    direction LR
    VT@{ icon: "tech:mysql", label: "Vitess · MySQL<br/>keyspaces by channel or user" }
    MC@{ icon: "sd:cache", label: "Memcached · Mcrouter<br/>Mcrib control plane" }
    CON@{ icon: "sd:network", label: "Consul<br/>discovery, hash rings" }
  end
  subgraph L5["Layer 5 · Async and search"]
    direction LR
    KF@{ icon: "tech:kafka", label: "Kafkagate · Kafka<br/>durable job buffer" }
    RQ@{ icon: "tech:redis", label: "JQRelay · Redis<br/>queues, worker pools" }
    SR@{ icon: "sd:search", label: "Search · Solr<br/>indexes, ranking" }
    PUSH@{ icon: "sd:bell", label: "Push service<br/>APNs, FCM" }
  end
  APP ==> ENV
  ENV ==>|"websocket"| GS
  ENV ==>|"HTTPS API"| WEB
  APP -.->|"boot, queries"| FL
  FL -.-> WEB
  WEB ==> VT
  WEB --> MC
  WEB ==>|"after the write"| AS ==> CS
  CS ==>|"one copy per gateway"| GS
  GS --> PS
  CON -.-> CS
  WEB ==>|"jobs"| KF ==> RQ
  RQ --> PUSH
  RQ --> SR
  class APP client
  class ENV,WEB edge
  class VT,CON store
  class MC cache
  class KF,RQ queue
  class SR service
  class PUSH external

KEY INSIGHTS:

• The channel is the unit of routing, not the workspace. Every event is addressed to a channel ID (a conversation, a user, a workspace, an org, a file); the channel server that owns the ID orders and fans out its events, and one workspace's channels are spread over every channel server
• Store, then send, then do the rest. A message is written to the database before any client sees it; everything that isn't the message itself (notifications, unfurls, indexing) is a job that can lag or retry without touching the send
• Two hops of fan-out, both by subscription. Channel servers send one copy per subscribed gateway server; gateway servers send one copy per subscribed client. Nothing in the core knows about individual websockets

Data Flow: Meera's @here#

%% title: Meera's @here, from send to 900 members
sequenceDiagram
  autonumber
  participant M as Meera's client
  participant E as Envoy
  participant W as Webapp
  participant V as Vitess
  participant A as Admin server
  participant C as Channel server
  participant G as Gateway servers
  participant O as Members' clients
  M->>E: POST message to #inc-payments, @here
  E->>W: route to the core region
  W->>W: auth, membership, may she notify @here
  W->>V: insert, routed by channel id
  V-->>W: stored
  W-->>M: ok, message id
  W->>A: message for channel C042
  A->>C: hash(C042), the owning channel server
  C->>G: one copy per subscribed gateway server
  G->>O: websocket push to subscribed clients
  Note over W,O: jobs follow: notifications, unfurls, index update

MEANWHILE, IN PARALLEL:

• The webapp enqueues jobs through Kafkagate: notification decisions for the 900 members, an unfurl of the dashboard link in the message, and our search index update; workers pick them up from Redis within seconds
• Kaveri Bank's engineer, typing in the shared channel, sends a typing event straight over her websocket to her gateway server, which routes it by channel ID to the channel server; it is never written to the database

Deep Dive 1: Workspaces, Orgs and Shared Channels: One Copy of Every Channel (Week 9 Concepts: Tenant Isolation; Week 1: Partitioning)#

This builds on Week 9, Day 1 · Tenant Isolation, Week 1, Day 1 · Partitioning Deep Dive and Week 9, Day 3 · Data Residency.

What is published: in 2019 Slack described the original model, in which a new workspace "was assigned to a specific database shard, messaging server shard, and search service shard". With "more than a billion messages" a week, copying shared channels to both sides was rejected: each channel has "a single channels row" on the originating workspace's shard, with all its messages, reactions and pins, plus "two additional shared_channels rows: one row for each workspace" whose overrides let "each workspace ... set its own name, topic and purpose". Privacy moved from G-prefixed IDs in a separate groups table to an explicit is_private flag, and the conversations.* API replaced channels.* and groups.*. Enterprise Grid (2017) added an org above workspaces, with cross-workspace channels on the org's shard, so the backend queries "both the workspace shard and, if absent there, on the org shard". Some of Slack's largest customers "have thousands of workspaces" (2024). Unified Grid, completed in March 2024, shows Grid users every workspace at once; where an API still needs workspace context, the backend falls back to iterating over the user's "relevant" workspaces, capped at 50. A Slack Connect channel can host up to 250 organisations, and only the owning one can invite or remove others (Slack help centre).

%% title: Resolving one channel for one user, across workspaces and companies
flowchart TD
  R["Request<br/>user, workspace, channel C"]:::service
  OWN@{ icon: "sd:database", label: "Channel row<br/>with the owner, one copy" }
  SIDE{"Same workspace<br/>as the owner?"}
  SC@{ icon: "sd:database", label: "shared_channels row<br/>this side's overrides" }
  VIEW["This side's view<br/>name, topic, is_private"]:::core
  MEM{"User a member,<br/>or allowed to join?"}
  OK["Serve the one history<br/>routed by channel id"]:::ok
  NO["Channel not found"]:::bad
  R ==> OWN ==> SIDE
  SIDE -->|"yes"| VIEW
  SIDE ==>|"no"| SC ==> VIEW
  VIEW ==> MEM
  MEM ==>|"yes"| OK
  MEM -.->|"no"| NO
  class OWN,SC store

• The tenant is a policy boundary, not a storage boundary. Admins, retention, data residency and keys still belong to an organisation; where a channel's rows sit is a routing detail. Checking "same workspace?" as a proxy for "allowed?" is exactly the shortcut shared channels break, so every access check asks about membership.
• One copy means one order. A shared channel has one history, one owner shard and one channel server, so both companies see messages in the same order. Two copies would need cross-database ordering that nobody wants to build.
• Org-wide data is fetched once. Slack's 2024 post notes it changed the real-time stack "to remove the need to fan-out org-wide data to every workspace on the grid". An org-level object (an org-wide channel, an org admin's profile) is one channel ID, subscribed to once, not copied into each of thousands of workspaces.
• Policy follows the organisation. Slack listed data retention and Enterprise Key Management as the hard remaining work for shared channels in 2019. With one copy, a design must say whose policy governs it; ours records the owning organisation on the channel and applies each side's retention and export rules to its own view.

# tenancy/shared_channels.py

"""
One copy of a shared channel (illustrative, after Slack's 2019 design). The channel
row and its history live with the originating side; every other workspace in the
channel has a shared_channels row that points at it and overrides name, topic and
privacy for that side. Membership, never workspace, decides access.
"""

from dataclasses import dataclass


@dataclass
class ChannelView:
    channel_id: str
    owner_workspace: str
    name: str
    is_private: bool


class NotFound(Exception):
    pass


def channel_view(db, workspace_id: str, channel_id: str) -> ChannelView:
    ch = db.channels.get(channel_id)                         # the single channels row
    if ch is None:
        raise NotFound(channel_id)
    if ch.workspace_id == workspace_id:
        return ChannelView(channel_id, ch.workspace_id, ch.name, ch.is_private)
    sc = db.shared_channels.get(workspace_id, channel_id)    # this side's pointer
    if sc is None:
        raise NotFound(channel_id)                           # same answer as "doesn't exist"
    return ChannelView(channel_id, ch.workspace_id,
                       sc.name or ch.name,
                       ch.is_private if sc.is_private is None else sc.is_private)


def read_history(db, user, workspace_id: str, channel_id: str, before_ts: str, limit: int = 50):
    view = channel_view(db, workspace_id, channel_id)
    if not db.members.contains(channel_id, user.id):
        if view.is_private or not db.can_join(user, view):
            raise NotFound(channel_id)
    return db.messages.latest(channel_id, before_ts, limit)  # routed by channel id, one copy

Deep Dive 2: Real Time: Channel Servers, Gateway Servers and the Edge Cache (Week 1 Concepts: Partitioning; Week 4: Multi-Tier Caching, Thundering Herd)#

This builds on Week 1, Day 1 · Partitioning Deep Dive, Week 4, Day 5 · Multi-Tier Caching and Week 4, Day 3 · Thundering Herd. For one process per community and its relays, see Bonus Problem 31 (Discord).

What is published: Slack's 2023 post describes four Java services. Channel servers "are stateful and in-memory, holding some amount of history of channels", each "mapped to a subset of channels based on consistent hashing", about 16 million per host at peak, where a channel ID can belong to "a user, team, enterprise, file, huddle, or a regular Slack channel". CHARMs manage the ring. Gateway servers "hold users' information and websocket channel subscriptions", run in "multiple geographical regions", and can be drained region by region. Admin servers are stateless, between the webapp and channel servers; presence servers have users hashed to them. A message goes webapp → admin server → channel server → "every GS across the world that is subscribed to that channel" → every subscribed client. Traffic peaks at 11 am and 2 pm local time, with spikes at the top of the hour from reminders and scheduled messages. Slack's 2017 Flannel post describes the problem before it: rtm.start sent "a full snapshot of a team's data", and teams of "tens of thousands of users" suffered slow boots and "reconnection storms". Flannel is an edge cache "deployed to our edge points-of-presence" that returns "a slimmed down version" of the startup data, answers queries such as Quick Switcher autocomplete, routes "users on the same team who are from the same networking region" to the same instance, and sends a mentioned user's profile to clients that lack it "just before sending the message". It served 4 million connections and 600,000 queries a second; lazy loading cut boot payloads 7× on a 1,500-user team and 44× on a 32,000-user team; moving presence to subscriptions cut presence events 5×.

%% title: One message, two hops of fan-out, three regions (illustrative)
flowchart TD
  W@{ icon: "sd:api", label: "Webapp<br/>message stored" }
  A["Admin server<br/>stateless, reads the ring"]:::service
  RING@{ icon: "sd:network", label: "Consul<br/>ring from CHARMs" }
  C["Channel server 17<br/>owns C042, sequence 9,312"]:::hot
  G1["Gateway · region A<br/>412 subscribed clients"]:::core
  G2["Gateway · region B<br/>36 subscribed clients"]:::core
  G3["Gateway · region C<br/>3 subscribed clients"]:::core
  U@{ icon: "sd:users", label: "Clients<br/>one websocket each" }
  W ==> A
  RING -.-> A
  A ==>|"hash(C042)"| C
  C ==> G1
  C ==> G2
  C --> G3
  G1 ==> U
  G2 ==> U
  G3 --> U
  class W edge
  class RING store
  class U client

• Hash the channel, not the tenant. A huge company's load lands on every channel server a little, instead of on one server a lot; a channel server failure is a 20-second delay for a sliver of everyone. The tenant boundary is enforced when subscriptions are built (only channels the user belongs to), not by where channels are placed.
• One copy per gateway server. The channel server's cost per message is the number of subscribed gateway servers, a handful; each gateway sends to its own clients. In 2017 the message server still forwarded "a copy to Flannel for each connected user", and Slack wrote that it's "much more efficient to send only one copy to each Flannel server".
• Order per channel, nowhere else. The owning channel server is the one place a channel's events pass through, so it can stamp a per-channel sequence. Nobody needs an order across channels, which is what lets channels sit on different servers.
• Transient events take the short path. Slack's "transient events", such as typing, go client → gateway → channel server → gateways and are never stored, so our design drops them under load before any message.
• Boot cost should track the screen, not the company. A slim boot is the user, their channel list with unread state, and one channel's messages with the users mentioned in them; everything else is a query answered at the edge. Team affinity means Paylane's 6,000 people share one cached copy per region, and on a cold morning the core sees only the first boot per workspace per region.
• Push what's about to be needed. Sending a mentioned user's profile just ahead of the message saves every recipient a round trip at the same moment: a herd avoided rather than absorbed.

# realtime/channel_server.py

"""
Channel IDs on a consistent-hash ring (illustrative). The owner stamps a per-channel
sequence and sends one copy per subscribed gateway server; gateways resubscribe only
the channels whose owner changed, and tell clients to fetch any gap from history.
"""

import bisect
import hashlib


def h(key: str) -> int:
    return int.from_bytes(hashlib.md5(key.encode()).digest()[:8], "big")


class Ring:
    def __init__(self, servers, vnodes=200):
        self.points = sorted((h(f"{s}#{i}"), s) for s in servers for i in range(vnodes))
        self.keys = [p for p, _ in self.points]

    def owner(self, channel_id: str) -> str:
        return self.points[bisect.bisect(self.keys, h(channel_id)) % len(self.points)][1]


class ChannelServer:
    def __init__(self):
        self.seq, self.subs = {}, {}          # channel_id -> last sequence / gateway ids

    def publish(self, channel_id, event, send):
        self.seq[channel_id] = self.seq.get(channel_id, 0) + 1
        event = {**event, "channel": channel_id, "seq": self.seq[channel_id]}
        for gw in self.subs.get(channel_id, ()):              # a handful, not thousands
            send(gw, event)


def on_ring_change(gateway, old: Ring, new: Ring, subscribe):
    for c, sockets in gateway.sockets.items():
        if new.owner(c) != old.owner(c):                      # only the dead server's arc moves
            subscribe(new.owner(c), c, gateway.id)
            for ws in sockets:
                ws.send({"type": "resync", "channel": c})     # clients fetch any gap from history

Deep Dive 3: Vitess: From Workspace Shards to Keyspaces by Channel (Week 1 Concepts: Partitioning, Hot Keys; Week 4: Caching Patterns)#

This builds on Week 1, Day 1 · Partitioning Deep Dive, Week 1, Day 4 · Hot Keys and Skew and Week 4, Day 1 · Caching Patterns. The February 2022 incident below is the war story Slack's new cache manager worked too well.

What is published: in 2020 Slack described its original layout, data "partitioned and scaled horizontally by workspace id", each shard at least two MySQL hosts in different data centres, all taking writes, and its limits: large customers hit "the largest available hardware", left "a few hot spots", and Enterprise Grid and Slack Connect broke "the paradigm that all data for a team will be on the same database shard". From 2017 it moved to Vitess, rejecting NoSQL and NewSQL stores, table by table with double writes, a backfill and "a parallel double-read diffing system": 99% of query load by December 2020, 2.3 million queries a second, median 2 ms, p99 11 ms, "dozens of keyspaces". Slack set out to shard message data "by the unique id of the channel"; by 2024 "the messages table is now sharded by channel ID". In March 2020 query rates rose 50% in a week, and a busy keyspace was split online. On 22 February 2022 a keyspace of channel membership "sharded by user" was overloaded by a group-DM membership query that, uncached, had to "query every shard", while a Consul agent upgrade on 25% of the fleet restarted the agents on cache nodes, which dropped out of service discovery; the Mcrib control plane swapped in empty spares and flushed nodes that rejoined. Slack throttled boots, read "only the data that was missing from memcached", used replicas, and later moved the query to a table "sharded by channel".

%% title: One query router, keyspaces sharded by what the queries ask for
flowchart TD
  APP@{ icon: "sd:api", label: "Webapp and services<br/>one SQL interface" }
  MC@{ icon: "sd:cache", label: "Memcached via Mcrouter<br/>cache-aside, long TTLs" }
  VG["Vitess query router<br/>sharding key to shard"]:::core
  MSG@{ icon: "tech:mysql", label: "Messages keyspace<br/>sharded by channel id" }
  MBU@{ icon: "tech:mysql", label: "Membership by user<br/>my channels" }
  MBC@{ icon: "tech:mysql", label: "Membership by channel<br/>members of C" }
  SC["Scatter query<br/>every shard, merge"]:::bad
  APP --> MC
  APP ==> VG
  VG ==>|"channel id known"| MSG
  VG ==>|"user id known"| MBU
  VG ==>|"channel id known"| MBC
  VG -.->|"no sharding key"| SC
  class APP edge
  class MC cache
  class MSG,MBU,MBC store

• Shard by the query's key. Messages by channel, so a channel's history is one shard and a big customer's channels spread over all of them. Membership twice, by user and by channel. Each table's key is chosen from its heaviest query, not from who the customer is.
• Treat a scatter query as a smell. It's correct, and at low volume it's fine; its cost grows with the number of shards, so it gets worse as you scale out. Keep a list of scatter queries, and give every high-volume one a table with the right key.
• A cache the database can't live without is capacity. If the database can only serve the boot load at a high hit rate, then anything that empties the cache (a node replaced, a flush, a restart wave) is a capacity loss. Change cache membership slowly, warm replacement nodes before they serve, and fetch only the missing keys on a miss.
• Bound the feedback loop. When timeouts stop the cache refilling, more load won't fix it. Shed new work (boots) until queries succeed again, then restore it in small steps.

# data/membership.py

"""
Reading group-DM membership through a cache (illustrative). Membership is written
twice, by user and by channel; reads use the by-channel table, fetch only what the
cache is missing, and may come from replicas because group-DM membership never changes.
"""

TTL_S = 7 * 24 * 3600            # immutable data: a long TTL


def members_of(cache, db, channel_ids: list) -> dict:
    keys = {c: f"gdm_members:{c}" for c in channel_ids}
    hit = cache.get_many(list(keys.values()))
    out = {c: hit[k] for c, k in keys.items() if k in hit}
    missing = [c for c in channel_ids if c not in out]
    if missing:                                              # only the misses, never "all again"
        rows = db.query(
            "SELECT channel_id, user_id FROM members_by_channel WHERE channel_id IN %s",
            missing, keyspace="membership_by_channel", replica_ok=True)   # one shard per channel
        for c in missing:
            out[c] = [r.user_id for r in rows if r.channel_id == c]
            cache.set(keys[c], out[c], ttl=TTL_S)
    return out

Deep Dive 4: The Job Queue and Notifications: One Message, Thousands of Side Effects (Week 3 Concepts: Queue vs Stream, Backpressure; Week 6: Notification Flow)#

This builds on Week 3, Day 1 · Queue vs Stream, Week 3, Day 3 · Backpressure and Flow Control and Week 6, Day 2 · Core Notification Flow.

What is published: Slack's 2017 job queue post: the queue handles "every Slack message post, push notification, URL unfurl, calendar reminder, and billing calculation", 1.4 billion jobs on the busiest days at up to 33,000 a second. An earlier outage began with database contention that slowed jobs until Redis hit "its maximum configured memory limit"; because dequeuing "required a bit of free Redis memory", the queue "remained locked up". Slack put Kafka in front of Redis rather than replacing it: Kafkagate (Go) writes jobs to Kafka waiting only for the leader's acknowledgement, a "bias towards availability over consistency"; JQRelay (Go) holds a Consul lock per topic, advances offsets only after a job reaches Redis, re-enqueues failing jobs to Kafka, and obeys rate limits from Consul. The cluster: 16 brokers, 32 partitions per topic, replication 3, 2 days of retention; the rollout double-wrote in "shadow" mode and sent heartbeat canaries through all 1,600 partitions every minute. For notifications, a 2023 post shows the path "through the webapp ..., job queue, push service, and several third-party services"; a 2017 post, building a thin desktop client for background workspaces, moved the logic for "whether to notify a user about a message" to the server, which computes has_unreads and mention_count and sends a desktop_notification event over the websocket. Backend requests are traced at 1%, but each notification is its own trace at 100%, linked to the message by span links, since one @here or @channel would otherwise make "billions of spans"; support triaged notification tickets 30% faster. Slack's help centre: @here notifies only active members, neither mention notifies people whose notifications are paused or inside a thread, and in Enterprise orgs only owners and admins can use them in channels of 10,000+ members.

%% title: The job path: a durable log in front of the working queue
flowchart TD
  WEB@{ icon: "sd:api", label: "Webapp<br/>message stored, jobs created" }
  KG["Kafkagate · Go<br/>HTTP in, same AZ, leader ack"]:::service
  K@{ icon: "tech:kafka", label: "Kafka<br/>16 brokers, 32 partitions, 2 days" }
  JR["JQRelay · Go<br/>one per topic, rate-limited"]:::core
  R@{ icon: "tech:redis", label: "Redis<br/>pending and in-flight lists" }
  WK@{ icon: "sd:worker", label: "Worker pools<br/>notify, unfurl, index, remind" }
  P@{ icon: "sd:bell", label: "Push service<br/>per device" }
  APN@{ icon: "tech:apple", label: "APNs<br/>iOS, macOS" }
  FCM@{ icon: "tech:firebase", label: "FCM<br/>Android" }
  WEB ==> KG ==> K
  K ==>|"offset advances after write"| JR
  JR ==> R ==> WK
  JR -.->|"job error, back to Kafka"| K
  WK ==> P
  P --> APN
  P --> FCM
  class WEB edge
  class K,R queue
  class WK service
  class P service
  class APN,FCM external

• A log absorbs; a queue schedules. Kafka holds days of jobs on disk and never refuses a write because workers are slow; Redis holds only what workers can take soon, and does the dedupe, retry and in-flight tracking the application already relies on. Adding the log in front changed nothing for the code that enqueues or runs jobs: Slack's "minimum viable change".
• The relay is the throttle. The rate from Kafka into Redis is a dial. When workers or the database are struggling, turn it down; the backlog waits safely in Kafka, and Redis memory stays flat. Slack also noted the next step: tying the relay's rate to Redis memory automatically.
• Choose the acknowledgement per job type. Waiting only for the leader keeps a send fast but can lose a job if a broker dies before replicating; for a link unfurl that's acceptable. Our design waits for in-sync replicas on job types where a loss is visible (a reminder, a billing step) and keeps leader acks for the rest.
• Separate the decision from the delivery. One job per message decides who to notify (preferences, DND, mute, which device); one job per device does the push. The fan-out job is cheap and fast; the pushes are many, slow and allowed to retry.

# notifications/decide.py

"""
Deciding who to notify for one message, and on which device (illustrative). Runs as
a job after the send. One decision job per message, one push job per device; each
notification gets its own id, used as its trace id and as the push's idempotency key.
"""

import uuid


def decide(msg, members, prefs, presence, enqueue):
    for m in members:
        if m.user_id == msg.author_id:
            continue
        p = prefs.get(m.user_id, msg.channel_id)          # per user, per channel
        mentioned = m.user_id in msg.user_mentions
        broadcast = (msg.here and presence.active(m.user_id)) or msg.channel_all
        if broadcast and msg.in_thread and not mentioned:
            continue                                      # @here and @channel don't notify in threads
        if p.muted and not mentioned:
            continue
        if not (mentioned or broadcast or p.level == "all" or msg.is_dm):
            continue
        if p.dnd_now() and not p.dnd_override_for(msg.author_id):
            continue
        nid = str(uuid.uuid4())                           # trace id for this one notification
        device = presence.foreground_device(m.user_id)    # desktop if she's at it, else phones
        if device == "desktop":
            enqueue("desktop_notification", user=m.user_id, msg=msg.id, nid=nid)
        else:
            for d in presence.push_devices(m.user_id):
                enqueue("push", user=m.user_id, device=d, msg=msg.id, nid=nid,
                        idempotency_key=f"{nid}:{d}")

Deep Dive 5: Search: A Searchable Log, Scoped to What Each Person Can Read (Week 7 Concepts: Indexing Pipeline, Query Processing and Relevance)#

This builds on Week 7, Day 2 · Indexing Pipeline and Week 7, Day 3 · Query Processing and Relevance.

What is published: "Slack" is a backronym for "Searchable Log of all Communication and Knowledge". In 2017 Slack wrote that "Solr powers search at Slack", that each user "has access to a unique set of documents" and queries "are rarely repeated". Recent sorts matches newest first; Relevant, used "about 17% of the time", became two-stage: Solr retrieves by cheap features, then the application re-ranks with an SVM trained on a pairwise transform of clicks, oversampling lower positions because a result at position n was "30% more likely to be clicked" than at n+1. The re-ranker raised clicked searches 9% and clicks at position 1 27%; Top Results runs Relevant alongside every Recent search. A 2017 talk by a Slack data engineer described "Solr offline indexing at scale" with a lambda architecture; a 2019 Slack talk showed the offline pipeline, separate encrypted indexes for Enterprise Key Management customers, and queries tokenised in all 8 supported languages. In 2026 Slack wrote that the pipeline "builds Solr search indexes from terabytes of data daily". Enterprise search (2025) adds external sources without storing them, querying their search APIs live with the user's own OAuth read scopes.

%% title: Two paths into the index, one permission-scoped query out
flowchart TD
  MSG@{ icon: "sd:api", label: "Message stored<br/>index job enqueued" }
  RT["Real-time indexer<br/>seconds behind"]:::service
  DW@{ icon: "sd:database", label: "Full data<br/>database and warehouse" }
  OFF@{ icon: "tech:spark", label: "Offline pipeline<br/>join, analyse, build daily" }
  IDX@{ icon: "sd:search", label: "Solr collections<br/>per tenant, EKM apart" }
  Q["Query service<br/>readable channel ids"]:::core
  RR["Re-ranker<br/>work-graph features"]:::hot
  U@{ icon: "sd:user", label: "Meera<br/>settlement file timeout" }
  MSG ==> RT ==> IDX
  DW ==> OFF ==>|"swap in rebuilt index"| IDX
  U ==> Q
  Q ==>|"filter by channel ids"| IDX
  IDX ==>|"top few hundred"| RR
  RR ==>|"re-check, Top Results"| U
  class MSG edge
  class DW store
  class OFF service
  class IDX store
  class U client

• Filter inside the engine. Fetching results and then dropping unreadable ones breaks pagination and leaks timing; passing the readable channel IDs as a filter means the engine only ever scores permitted documents. A final membership check on the returned page catches anything that changed in the last few seconds.
• Real time for freshness, batch for truth. The real-time path makes a new message findable in seconds; the batch rebuild re-derives every document from the source data, so a new analyser or feature applies to all history and any message the real-time path missed comes back. Both write the same schema, so swapping a rebuilt index in is invisible.
• Retrieve cheap, rank rich. Solr scores thousands of matches by text match and a few stored fields; the re-ranker scores only the top few hundred with features that need the user's context (affinity to the author, the channel's priority for her, reactions, pins).
• Federate what you mustn't store. External documents stay in their systems: search them live with the user's own credentials, so permissions and freshness are the source system's, not a copy's.

# search/query.py

"""
A permission-scoped, two-stage search (illustrative). Readable channel ids go into
the engine as a filter; the engine returns a few hundred candidates by cheap
features; the re-ranker scores them with the searcher's work-graph features; the
returned page is re-checked against membership.
"""

CANDIDATES = 300


def search(user, text, membership, engine, graph, model, sort="recent", page=20):
    readable = membership.readable_channel_ids(user)          # computed now, not cached for long
    partitions = membership.index_partitions(readable)        # own tenant plus shared-channel owners
    hits = engine.query(text, partitions=partitions,
                        filter_channels=readable,
                        sort="date desc" if sort == "recent" else "score desc",
                        rows=CANDIDATES if sort == "relevant" else page)
    if sort == "relevant":
        feats = [[h.lucene_score, h.age_days,
                  graph.affinity(user.id, h.author_id),
                  graph.channel_priority(user.id, h.channel_id),
                  h.reactions, h.pinned, h.author_id == user.id] for h in hits]
        scores = model.score(feats)
        hits = [h for _, h in sorted(zip(scores, hits), key=lambda p: -p[0])]
    final = [h for h in hits[:page * 2] if membership.is_member(user.id, h.channel_id)]
    return final[:page]

Meera's Monday#

Edge Cases#

What is published: Slack's message-send success rate "is usually over 99.999%"; on 4 January 2021 an external monitor paged first, and Slack's own dashboards and alerts failed because they "were running in a different VPC from their backend databases", a dependency Slack said it would remove by running them in the same VPC. The job queue rollout sent heartbeat canaries through every Kafka partition each minute and alerted on their end-to-end timing. Notification traces flow to Elasticsearch, Grafana and the warehouse, where client engineers built dashboards and data scientists built funnels of notification open rates. Backend requests are traced at 1%. During the June 2021 incident, "nearly every graph we had aggregated by target AZ" showed the bad zone, which is what made a drain button the obvious fix.

Monitoring Dashboard#

SLOs for the Platform#

The interviewer leans back. "This one is from Slack's own postmortem. Tell me what you'd have done at the time, and what you'd change."

Summary: Concepts Applied from 10-Week Course#

Why Slack Matters#

Self-Assessment Checklist#

After studying this case study, you should be able to:

Tenancy and real time:

Edge, storage and jobs:

Notifications, search and operations:

Sources#

Slack Engineering (slack.engineering):

• Blog

  Sameera Thangudu — Real-time Messaging (channel, gateway, admin and presence servers; CHARMs; 16 million channels per host; under 20 s replacement; 500 ms), 11 April 2023: https://slack.engineering/real-time-messaging/

• Blog

  Bing Wei — Flannel: An Application-Level Edge Cache to Make Slack Scale (rtm.start; team affinity; 4 million connections; 7× and 44× smaller boots; presence 5×), 31 May 2017: https://slack.engineering/flannel-an-application-level-edge-cache-to-make-slack-scale/

• Blog

  Mark Christian — Getting to Slack faster with incremental boot, 5 July 2016: https://slack.engineering/getting-to-slack-faster-with-incremental-boot/

• Blog

  Yingyu Sun and Mike Demmer — How Slack Built Shared Channels (workspace shards; a billion messages a week; shared_channels rows; is_private; 160,000+ users, 5,000+ shared channels), 16 September 2019: https://slack.engineering/how-slack-built-shared-channels/

• Book

  Ian Hoffman and Mike Demmer — Unified Grid: How We Re-Architected Slack for Our Largest Customers (org shards; messages sharded by channel ID; 50 relevant workspaces; March 2024), 31 July 2024: https://slack.engineering/unified-grid-how-we-re-architected-slack-for-our-largest-customers/

• Blog

  Arka Ganguli, Guido Iaquinti, Maggie Zhou and Rafael Chacón — Scaling Datastores at Slack with Vitess (2.3 million QPS, 2 ms median, 11 ms p99; migration; March 2020 growth; six regions), 1 December 2020: https://slack.engineering/scaling-datastores-at-slack-with-vitess/

• Blog

  Saroj Yadav, Matthew Smillie, Mike Demmer and Tyler Johnson — Scaling Slack's Job Queue (1.4 billion jobs on the busiest days, 33,000 a second; Kafkagate, Kafka, JQRelay; cluster and rollout), 6 December 2017: https://slack.engineering/scaling-slacks-job-queue/

• Blog

  Johnny Rodgers, Charlie Hess, Raissa Largman, Jamie Scheinblum and Chris Sullivan — Reducing Slack's memory footprint (server-side notification logic; desktop_notification), 3 March 2017: https://slack.engineering/reducing-slacks-memory-footprint/

• Blog

  Suman Karumuri and George Luong — Tracing Notifications (notification path; one trace per notification; 30%), 4 April 2023: https://slack.engineering/tracing-notifications/

• Blog

  Isabella Tromba, John Gallagher and Jason Liszka — Search at Slack (Solr; Recent and Relevant; two-stage ranking; 9% and 27%), 7 February 2017: https://slack.engineering/search-at-slack/

• Blog

  Ian Hoffman and Claire Bowman — How we built enterprise search to be secure and private (the backronym; federated external search), 7 March 2025: https://slack.engineering/how-we-built-enterprise-search-to-be-secure-and-private/

• Blog

  Mahendran Vasagam — From SSH to REST: A Security-Driven Modernization of Slack's EMR Data Pipelines (Solr indexes from terabytes daily), 5 May 2026: https://slack.engineering/from-ssh-to-rest-a-security-driven-modernization-of-slacks-emr-data-pipelines/

• Blog

  Ariane van der Steldt and Radha Kumari — Migrating Millions of Concurrent Websockets to Envoy (panic routing on 4 January 2021), 15 March 2021: https://slack.engineering/migrating-millions-of-concurrent-websockets-to-envoy/

• Blog

  Laura Nolan — A Terrible, Horrible, No-Good, Very Bad Day at Slack (12 May 2020), 29 June 2020: https://slack.engineering/a-terrible-horrible-no-good-very-bad-day-at-slack/

• Blog

  Laura Nolan — Slack's Outage on January 4th 2021, 1 February 2021: https://slack.engineering/slacks-outage-on-january-4th-2021/

• Blog

  Laura Nolan, with Glen D. Sanford, Jamie Scheinblum and Chris Sullivan — Slack's Incident on 2-22-22, 26 April 2022: https://slack.engineering/slacks-incident-on-2-22-22/

• Blog

  Cooper Bethea — Slack's Migration to a Cellular Architecture (30 June 2021; siloing; zone drains), 22 August 2023: https://slack.engineering/slacks-migration-to-a-cellular-architecture/

Slack company pages, help centre and talks:

• Read

  Slack — Slack CEO Stewart Butterfield Shares Updated Business Metrics (12.5 million simultaneously connected users; messages per user per day up ~20% since 1 February), Business Wire, 26 March 2020 (archived copy): https://web.archive.org/web/2020/https://www.businesswire.com/news/home/20200326005225/en/Slack-CEO-Stewart-Butterfield-Shares-Updated-Business

• Blog

  Slack Help Centre — Slack Connect guide (up to 250 organisations; owner's controls): https://slack.com/help/articles/115004151203-Slack-Connect-guide--Work-with-external-organizations

• Blog

  Slack Help Centre — Notify a channel or workspace (@here and @channel rules): https://slack.com/help/articles/202009646-Notify-a-channel-or-workspace

• Read

  Ananth Packkildurai (Slack) — Search Infrastructure @ Slack using Lambda Architecture, The Fifth Elephant 2017: https://hasgeek.com/fifthelephant/2017/sub/search-infrastructure-slack-using-lambda-architect-ECTbWgMBsnyq6jVkBYM28E

• Paper

  Slack Search & Discovery — Search at Slack (offline indexing; EKM indexes; 8 languages), Berlin Buzzwords 2019 slides: https://2019.berlinbuzzwords.de/sites/2019.berlinbuzzwords.de/files/media/documents/slack_search_-_berlin_buzzwords_2019.pdf

Design note: the architecture beyond what Slack's posts describe, the code, thresholds, dashboards and every number labelled as an assumption or estimate are an illustrative design, not Slack's internals. Today's connected clients, message rates, events and recipients per message, websockets per gateway server, distinct channel IDs, job sizes, the notification rules in the code, search freshness and partitioning details, and all of Paylane's numbers are assumptions. Meera Iyer, Paylane and Kaveri Bank are fictional. Slack's published numbers come from posts and releases between 2016 and 2026 and describe the system at the time of each.

Further Reading#

Related systems to study:

• Side quest

  Discord (Bonus Problem 31 in this series): one process per community server, relays and passive sessions, the opposite routing choice

  ↗(opens in new tab)
• Side quest

  WhatsApp (Bonus Problem 2): one-to-one and group messaging with offline delivery and end-to-end encryption

  ↗(opens in new tab)
• Side quest

  Gmail (Bonus Problem 28): another per-user searchable store, with spam instead of channels

  ↗(opens in new tab)
• Side quest

  GitHub (Bonus Problem 29): MySQL and Vitess for metadata, and Kafka events per push, at a code host

  ↗(opens in new tab)

From the war stories:

• Read

  Slack's first workday of 2021 overloaded a network hub that was meant to scale itself

• Read

  Slack's new cache manager worked too well, and a routine upgrade tipped Slack into overload

Foundational concepts (this course):

• Lesson

  Week 1, Day 1 — Partitioning Deep Dive

  ↗(opens in new tab)
• Lesson

  Week 1, Day 4 — Hot Keys and Skew

  ↗(opens in new tab)
• Lesson

  Week 3, Day 1 — Queue vs Stream

  ↗(opens in new tab)
• Lesson

  Week 3, Day 3 — Backpressure and Flow Control

  ↗(opens in new tab)
• Lesson

  Week 4, Day 1 — Caching Patterns

  ↗(opens in new tab)
• Lesson

  Week 4, Day 3 — Thundering Herd

  ↗(opens in new tab)
• Lesson

  Week 4, Day 5 — Multi-Tier Caching

  ↗(opens in new tab)
• Lesson

  Week 6, Day 2 — Core Notification Flow

  ↗(opens in new tab)
• Lesson

  Week 6, Day 4 — Scale, Reliability and Edge Cases

  ↗(opens in new tab)
• Lesson

  Week 7, Day 2 — Indexing Pipeline

  ↗(opens in new tab)
• Lesson

  Week 7, Day 3 — Query Processing and Relevance

  ↗(opens in new tab)
• Lesson

  Week 9, Day 1 — Tenant Isolation

  ↗(opens in new tab)
• Lesson

  Week 9, Day 3 — Data Residency

  ↗(opens in new tab)

"Route by the channel, govern by the organisation, and put everything a message causes on a queue that never says no."