Day 1Search Fundamentals & Architecture

Why LIKE fails. LIKE '%red%' can't use a B-tree, which is sorted by string start, so 50M rows get a full scan. It also has no relevance ranking, typo tolerance, word boundaries, stemming or synonyms.

The inverted index. Pre-compute word to documents instead of scanning documents for words. A query is one lookup per term plus an intersection, smallest list first: "running shoes" takes under a millisecond.

Text analysis. Character filters (strip HTML), then the tokenizer, then token filters (lowercase, stop words, stemming, synonyms). The same analyzer must run at index time and query time, or "Running" never matches the indexed "run".

The multi-field pattern. One source field, several representations: brand as text for search, brand.keyword for facets and sorting, brand.autocomplete for typeahead. IDs are keyword: they're only exact-matched.

Scatter-gather. A coordinating node sends the query to every shard (primary or replica). Each searches its own inverted index and returns its top N with scores; the coordinator merges, sorts by score and returns the overall top N.

The product search cluster. 20 primary shards plus 20 replicas on 6 data nodes, 3 dedicated master nodes, 2 coordinating nodes. Text goes in must to drive the score; category, price and stock go in filter.

• Different analyzers at index and query time, so tokens never match.
• Oversharding: per-shard memory and file-handle overhead degrades queries.
• Not planning primary shards: the count is fixed at creation, so changing it needs _split/_shrink or a reindex.

Day 2Indexing Pipeline

Never dual-write. Writing PostgreSQL then Elasticsearch breaks four ways: the ES write fails, the process crashes between writes, a partition holds the transaction open, or retries reorder updates so ES ends at $100 while PostgreSQL has $120.

Change data capture. App to PostgreSQL; Debezium reads the WAL through logical replication and publishes change events to Kafka; the indexing pipeline consumes them into Elasticsearch. The app writes to one place, and events replay within Kafka's retention.

Version by WAL position. Each bulk operation carries version: event.lsn with version_type: external. If an older change arrives late after retries or reordered batches, ES rejects it with a version conflict instead of overwriting newer data.

Batch, but by priority. Flush at 500 documents or after 5 seconds, so quiet periods don't strand updates. Stock going to zero is CRITICAL in 10-document batches; description edits are NORMAL at 500; bulk imports LOW at 1,000.

Zero-downtime reindex. Reads stay on the products alias over v1 while updates go to both indexes and old data is copied with op_type: create. Verify counts, switch the alias in one atomic remove+add, then delete v1.

• Running a big import through the live pipeline, starving CDC updates.
• Skipping index aliases, so a reindex can't switch with zero downtime.
• Blaming the pipeline when the ES document is right but a result cache is stale.

Day 3Query Processing & Relevance

BM25 in one line. Score grows with term frequency but saturates (k1, default 1.2), rare terms weigh more (IDF), and long documents are normalized (b, default 0.75). 10 occurrences aren't 10x better than 1.

Rare terms win. In the example corpus, "nike" (2% of documents) has IDF 3.912, "shoes" (10%) 2.3026, and "the" (90%) 0.1054. A match on the rarer term says more about relevance.

Query context vs filter context. What the user typed is scored in query context. What they selected (facets, toggles, ranges) goes in filter: yes or no, unscored, cacheable as a bitset. The lesson puts the split at ~3x faster.

Boost in layers. Start with field boosts (name^3, brand^2, description), add business boosts in should clauses, and use function_score (popularity, recency decay, rating, featured) only when needed. A/B test every ranking change.

Never return nothing. Progressive relaxation: exact AND query, then drop low-priority filters (rating, price), then fuzzy matching, then any-term matching, then no filters at all. Category stays: it's a strong intent signal.

Pick the multi_match type. best_fields with tie_breaker 0.3 is the product default. cross_fields lets "John" match first_name and "Smith" last_name. most_fields sums scores across analyses. phrase_prefix and bool_prefix serve autocomplete.

• Putting category, brand and price in must: they sway scores without adding relevance and aren't cached.
• Filtering out-of-stock items away when they should be demoted with a boosting query.
• Over-boosting, which makes relevance unpredictable.

Day 4Advanced Search Features

Edge n-gram autocomplete. Pre-compute prefixes at index time: "nike" is stored as "ni", "nik", "nike", so each keystroke is an exact token lookup. The search analyzer only lowercases, or "nik" would also match every "ni" word.

"Did you mean". A phrase suggester over word shingles, with collate and prune so only suggestions that find documents survive. Show it only when the correction has over 2x the original's results and more than 10.

Multi-select facets. User selections go in post_filter, which narrows hits but not aggregations. Each facet counts as if its own filter weren't applied: tick Nike and the brand facet still shows Adidas.

Synonyms, both kinds. Two-way groups (tv, television, telly) and one-way rules (ipod => ipod, mp3 player), so "mp3 player" isn't pulled to iPods. Query-time synonym_graph handles multi-word; updateable search analyzers reload without a reindex.

A field per language. One index with name_es, name_fr and so on, each with its own analyzer (spanish, kuromoji), because stemming, stop words and tokenization differ. Separate indexes separate cleanly but make cross-language search harder.

Personalize with boosts, not filters. Brand and category affinities from 30 days of behavior (purchases weigh 3, views 1) become function weights scaled by personalization_strength. Recent views get 0.8. Other brands still show, avoiding a filter bubble.

• N-gramming the query as well, so "nik" also matches every "ni" word.
• Letting a facet's own selection narrow its counts, hiding the other brands.
• Making brand synonyms two-way, pulling "mp3 player" searches to iPods.

Day 5Search Operations & Scale

Dedicated node roles. 3 dedicated master nodes give a quorum that survives one loss without split brain. Coordinating nodes route and aggregate, shielding data nodes; data nodes are memory-optimized with SSDs.

Size shards for growth. Primary count can't change later, so plan ahead: 620 GB of index × 1.5 growth is 930 GB; at 30 GB per shard that's ~31, rounded to 32 primaries, 64 shards with one replica.

The largest requirement wins. Count data nodes separately for storage, memory and CPU, take the largest (minimum 3) and round to even for AZ spread. At 50K QPS about 330 cores make CPU the limit: 6 data nodes become 22.

Pre-scale, don't reactive scale. New nodes need time to join and take over shards. Scale up 24 hours before Black Friday with an extra replica, peak at 22 data and 10 coordinating nodes, then scale down gradually.

Degrade gracefully. Warning (p99 > 500 ms or CPU > 70%): longer cache TTL, no spell check or personalization. Critical (p99 > 1 s or CPU > 85%): also no facets or highlighting, simpler queries. Better slow search than no search.

Snapshots plus CDC. Lose the cluster: provision a new one, restore the latest S3 snapshot, resume CDC so Kafka replays later changes. For corruption, restore beside the live index and swap the alias. Untested backups are not backups.

• Force-merging a live, still-written index to one segment: the huge segment piles up deleted docs.
• Aggregating on text fields, loading fielddata onto the heap; use keyword.
• Deep from offsets instead of search_after.

BossDesigning a Legal Document Search System

Size the corpus. 100M documents (50M public, 50M private) at ~65 KB is 6.5 TB of text. 2x index overhead and 2x for replicas make 26 TB in Elasticsearch, plus 50 TB of original PDFs in S3: ~80 TB.

Size the traffic. 50,000 concurrent users at 20 searches an hour is 1M an hour, ~280 a second; autocomplete at 5 keystrokes per search is ~1,400 a second. Today's 10,000 a minute (~170/s) leaves ~1.7x headroom.

A real query parser. A lexer and parser turn (breach /5 contract) AND damages NOT punitive into must, should and must_not clauses. Proximity becomes span_near with in_order: false; /s means ~15 words, /p ~50.

Isolation in layers. Per-tenant indices, a service-injected tenant_id filter, document-level security, a tenant taken from the validated JWT (never the request body), and audit logs. Zero tolerance means no single control is trusted alone.

Citations as edges. A separate index of {citing_id, cited_id, treatment} makes "who cites X" a term query on cited_id, with aggregations for treatment breakdowns, "good law" checks and 2-hop chains.

Triage every upload. Store the original in S3 first. A PDF with under 100 characters of text in its first pages is scanned: render each page at 300 DPI and OCR it. Word goes through Tika. Then publish to Kafka.

• Trusting one isolation control, or a tenant_id from the request body.
• Using a phrase query for proximity: "liability for negligence" should match negligence /10 liability.
• Fuzzy matching citation variants instead of a canonical case ID plus name synonyms.