Python Web Scraping Exercises: Build a Polite Pipeline

Start from a saved page

Download once during development, save the HTML, and parse the fixture repeatedly. This removes network variance from your debugging and lets you test edge cases such as missing attributes or repeated links.

from html.parser import HTMLParser

class Links(HTMLParser):
    def __init__(self):
        super().__init__()
        self.links = []
    def handle_starttag(self, tag, attrs):
        href = dict(attrs).get("href")
        if tag == "a" and href:
            self.links.append(href)

p = Links()
p.feed('<a href="/a">A</a><a href="/a">A</a><a href="/b">B</a>')
print(p.links)

A fixture you can rerun

Build the request separately

Keep request construction in one function. Set a descriptive User-Agent and a timeout when you eventually send it. The browser runner on this site does not send network requests, so these examples stop at the Request object.

from urllib.request import Request

request = Request(
    "https://example.com/docs",
    headers={"User-Agent": "LearningBot/1.0"},
)
print(request.full_url)
print(request.get_header("User-agent"))

No network in this example

Branch on the status

Do not parse a 404 page as if it were the page you wanted. A small status function keeps the decision visible, and it gives the retry policy one place to live.

def outcome(status):
    if 200 <= status < 300:
        return "parse"
    if 300 <= status < 400:
        return "follow redirect"
    if status == 429:
        return "back off"
    return "stop"

print(outcome(200))
print(outcome(429))

A small status policy

Store compact records

Extract the fields the task needs and leave the rest behind. Use a list of dictionaries for structured output, and deduplicate by a stable key such as the absolute URL. This keeps the result inspectable before it reaches a file or database.

rows = [
    {"url": "/a", "title": "A"},
    {"url": "/a", "title": "A"},
    {"url": "/b", "title": "B"},
]
unique = list({row["url"]: row for row in rows}.values())
print(unique)

One record per URL

Read the rules before the crawl

robots.txt is a starting point, not the whole agreement. Read the site terms, identify your client, limit request rate, and avoid personal data unless you have a lawful basis. If a site says no, stop.

robots = "User-agent: *\nDisallow: /private\nCrawl-delay: 2"
rules = {}
for line in robots.splitlines():
    if ":" in line:
        key, value = line.split(":", 1)
        rules[key.strip().lower()] = value.strip()
print(rules["disallow"])
print(rules["crawl-delay"])

The smallest useful check

Your practice checklist

• Save a real page once and test the parser against the fixture.
• Build requests with an identifiable User-Agent and a timeout.
• Decide what to do for 2xx, 3xx, 4xx, and 5xx responses.
• Store only the fields the task needs, keyed by a stable URL.
• Read robots.txt and site terms before scaling the crawl.

Common questions

Keep reading:

• Python HTTP requests for scraping
• Python Exercises With Answers: How to Practice Without Copying
• Python Practice for Beginners: A Simple 4-Week Plan