Web crawler: a billion pages, politely, without getting lost
One strong interview answer to "design a web crawler": a two-layer frontier where priority decides what to fetch and per-host queues decide when, URL normalisation and a 12 GB Bloom filter for the seen set, SimHash for near-duplicate pages, defences against spider traps, and the robots.txt and lease rules that keep it polite and robust.
01 · An afternoon's loop
IP-05 · Design a Web Crawler
Take a URL, fetch it, pull the links, repeat.
A loop, in an afternoon
A web crawler looks like a loop you could write in an afternoon.
0:00 / 13:10