Python pagination and deduplication

Pagination turns one page into many, and links can form cycles. A crawl needs a queue for what comes next, a visited set for what has already been seen, and a budget for when to stop. These are ordinary data structures, not scraping magic. You will also stop repeated next links and keep pagination on one origin.

Use a queue for the frontier

A deque is an efficient first-in, first-out queue: append adds to the right and popleft removes from the left. That order is breadth-first, so the crawler finishes nearby pages before going deeper. A list can work, but popleft is the operation deque is designed for.

from collections import deque

queue = deque(["/page/1", "/page/2"])
print(queue.popleft())
print(list(queue))

First in, first out

Remember what you have visited

A set makes membership checks fast. Add a URL after you remove it from the queue, and skip it if it is already present. Without this memory, two pages that link to each other can keep the crawler busy forever.

seen = set()
for url in ["/a", "/a", "/b"]:
    if url in seen:
        continue
    seen.add(url)
print(len(seen))

One visit per URL

Bound the crawl

Even a correct queue can run too long. Add a maximum number of pages, a maximum depth, or a wall-clock budget. Stop when the queue is empty or the budget is spent. Log why the crawl stopped so the result is not mistaken for complete coverage.

pages = {"/1": ["/2"], "/2": ["/3"], "/3": []}
queue = ["/1"]
seen = []
while queue and len(seen) < 2:
    url = queue.pop(0)
    if url in seen:
        continue
    seen.append(url)
    queue.extend(pages.get(url, []))
print(seen)

A small, explicit budget

Resolve before you compare

The same destination can appear as /docs, ../docs, and https://example.com/docs. Resolve relative links with urljoin, drop the fragment, and only then check the visited set. Normalize consistently or the set will fill with duplicates.

from urllib.parse import urljoin, urlsplit, urlunsplit

base = "https://example.com/docs/page.html"
raw = "../guide.html#top"
absolute = urljoin(base, raw)
parts = urlsplit(absolute)
clean = urlunsplit((parts.scheme, parts.netloc, parts.path, parts.query, ""))
print(clean)

Resolve, strip the fragment, compare

Stop when a next link repeats

Pagination loops are common when a site ignores an invalid page number or repeats the last page forever. Normalize each next URL and keep a seen set. Stop before fetching a URL that is already in the set, and combine this with a hard page budget.

from urllib.parse import urljoin, urlsplit, urlunsplit

def key(url):
    parts = urlsplit(url)
    return urlunsplit((parts.scheme, parts.netloc, parts.path, parts.query, ""))

start = "https://example.com/list"
links = ["/page/1", "/page/2", "/page/1"]
seen = set()
order = []
for link in links:
    url = urljoin(start, link)
    marker = key(url)
    if marker in seen:
        print("loop at", url)
        break
    seen.add(marker)
    order.append(url)
print(len(order), order)

A seen set ends the loop

Keep pagination on the same origin

A next link can point at another host, especially after an open redirect. Compare scheme, host, and port before adding it to the frontier. This is a scope decision, not a claim that other hosts are malicious; a crawler should stay inside the boundary its owner defined.

from urllib.parse import urljoin, urlsplit

base = "https://example.com/list?page=1"
base_parts = urlsplit(base)
for link in [
    "/list?page=2",
    "https://evil.example/list?page=3",
    "//example.com/list?page=4",
]:
    target = urljoin(base, link)
    parts = urlsplit(target)
    same = (parts.scheme, parts.hostname, parts.port) == (
        base_parts.scheme,
        base_parts.hostname,
        base_parts.port,
    )
    print(link, same, parts.netloc)

A frontier with a boundary

Try it yourself

Change the values, predict the output, then run the code. The Python runtime is downloaded the first time you press Run, so the first run can take a few seconds.