robots.txt says 404: crawl it all. 503: crawl nothing.

transcript

Your crawler asks a website for its rules. It gets an error. One error means crawl everything. Another means crawl nothing at all.

A polite crawler reads a site's robots.txt before fetching anything else. It says which paths are off limits. Disallow means stay out. And how fast to go. By default, one request a second per host.

But here's the catch. What if robots.txt itself fails? RFC 9309 answers it. And the answer depends on the error.

A 404 says the file isn't there. No rules. You may crawl. A 503, or a network error, says the site is unreachable. The rules might exist. You just can't see them. So assume everything is off limits. Not forever. Retry later, and crawl once you can read the rules.

The same instinct runs through the whole crawler. A host answers slowly? Back off. It answers 429, or 503? Back off again. A server dripping one byte a second? Timeouts, so it can't hold a fetcher hostage.

A crawler is a guest. Millions of servers never agreed to be crawled. Missing rules mean go. Unreadable rules mean stop.

Full breakdown in the lecture.