This website never ends, and it breaks no rules

transcript

This website never ends. Your crawler could fetch it forever. And it isn't breaking a single rule.

Your crawler fetches a calendar page. It has a next month link. A new URL. So it follows it. That page has a next month link too. Forever. Every URL is new. Every page is valid.

These are spider traps. The calendar is just one. A site that adds a new session ID to every link. Each one looks new. A server that generates infinite unique paths, on the fly.

Here's the catch. None of these break any rule. They just never end. So you put a limit on everything.

Cap the URL length and the path depth. Give each host a page budget per crawl cycle. The calendar runs out. Detect repeating path segments. A, B, A, B, A, B. And lower the priority of hosts whose new pages are mostly near-duplicates.

Each limit caps how far one trap can pull your crawler in. A crawler without limits isn't thorough. It's lost.

Full breakdown in the lecture.