A million URLs from one site: how a crawler stays fast and polite

transcript

Your crawler has a million URLs from one big site. So it sends every fetcher there. At once.

Assume a billion pages a month. About 400 pages a second. A fetch takes about a second. So 400 to 800 fetches in flight. But politeness says: at most one request a second, per host.

With one queue, those million URLs come out in a row. Every fetcher hits the same site. Fast, and the opposite of polite.

Split the job in two. First, score every URL into priority queues. That's what to fetch. Then route each URL into its host's own queue. One host per queue. A heap tracks each host's next allowed time. That's when to fetch. A fetcher pops the earliest host that's due, and fetches one URL. Then pushes the host back. Now plus a second, or its crawl delay.

Hundreds of fetches run in parallel. Spread across many hosts. And no single host sees more than one request per interval.

It's the design described for the Mercator crawler. Late 1990s. Still the standard answer. Priority decides what. Per-host queues decide when.

Full breakdown in the lecture.