Python robots.txt Exercises: Rules, Delays, and Ethics

Parse the directives

Each non-comment line is a key and a value separated by a colon. Keys are case-insensitive. Store the values you need, but do not pretend this tiny parser is a complete robots standard implementation.

robots = '''User-agent: *
Disallow: /private
Allow: /private/public
Crawl-delay: 2'''
rules = {}
for line in robots.splitlines():
    line = line.strip()
    if not line or line.startswith("#") or ":" not in line:
        continue
    key, value = line.split(":", 1)
    rules[key.strip().lower()] = value.strip()
print(rules)

Keys, values, comments

Use the longest matching rule

Allow and Disallow can overlap. The common convention is that the longest matching path wins; when lengths tie, the more specific rule usually decides. Do not simply stop at the first line.

rules = [("allow", "/private/public"), ("disallow", "/private")]
path = "/private/public/report"
decision = "allow"
best = -1
for kind, prefix in rules:
    if path.startswith(prefix) and len(prefix) > best:
        decision = kind
        best = len(prefix)
print(decision)

Specific beats broad

Turn delay into a schedule

Crawl-delay is a request-spacing hint, not a guarantee that the server is healthy. Record a timestamp after each request and wait until the next allowed time. If the site publishes a stricter limit elsewhere, follow the stricter one.

from datetime import datetime, timedelta

start = datetime(2026, 1, 1, 12, 0, 0)
delay = 2
times = [start + timedelta(seconds=delay * i) for i in range(3)]
print([t.strftime("%H:%M:%S") for t in times])

Space requests deliberately

Back off on 429

A 429 response means you are sending too many requests. Read Retry-After when it is present, wait at least that long, and reduce concurrency. Retrying immediately makes the problem worse and can get your client blocked.

status = 429
retry_after = "30"
if status == 429:
    wait = int(retry_after)
    print(f"wait {wait} seconds")
else:
    print("continue")

Respect the server's answer

Collect less, ask first

If a page contains names, emails, or health details, do not collect them just because they are visible. Define the minimum fields, avoid storing identifiers, and delete raw pages when they are no longer needed. Access controls, logins, CAPTCHAs, and paywalls are boundaries, not obstacles to bypass.

record = {"name": "Ada", "email": "ada@example.com", "bio": "Python"}
public = {key: value for key, value in record.items() if key != "email"}
print(public)

Minimize before storing

Your practice checklist

• Parse comments, blank lines, and key/value pairs separately.
• Apply the longest matching Allow or Disallow rule.
• Space requests by at least the published delay.
• Honor Retry-After and reduce concurrency on 429.
• Collect the minimum data and never bypass access controls.

Common questions

Keep reading:

• Python robots.txt and polite scraping
• Python Exercises With Answers: How to Practice Without Copying
• Python Practice for Beginners: A Simple 4-Week Plan