Python pagination and deduplication Exercises With Answers

Why keep a visited set during a crawl?

1. To sort the pages alphabetically
2. To avoid cycles and repeated pagescorrect
3. To store the HTML
4. To change the User-Agent

Answer: To avoid cycles and repeated pages

Why: A visited set remembers URLs that were already processed.

Which deque method removes the leftmost item?

1. pop()
2. removeleft()
3. shift()
4. popleft()correct

Answer: popleft()

Why: popleft removes from the front, which gives FIFO order for a queue.

Why set a maximum page count?

1. To bound the work and stop runaway crawlscorrect
2. To make URLs shorter
3. To remove query strings
4. To decode HTML entities

Answer: To bound the work and stop runaway crawls

Why: A page budget is a simple stop condition even when links keep appearing.

What does this code print?

from collections import deque
pages = {"/1": ["/2", "/a"], "/2": ["/b"]}
queue = deque(["/1"])
seen = []
while queue and len(seen) < 3:
    url = queue.popleft()
    if url in seen:
        continue
    seen.append(url)
    queue.extend(pages.get(url, []))
print(seen)

1. ['/1', '/a', '/2']
2. ['/1', '/2', '/a']correct
3. ['/1', '/2', '/a', '/b']
4. ['/2', '/1', '/a']

Answer: ['/1', '/2', '/a']

Why: The breadth-first queue visits /1, then its links /2 and /a, stopping at the budget of three.

Complete the queue type.

from collections import deque
queue = ____(["/1", "/2"])
print(queue.popleft())

Answer: deque

Why: deque supports efficient appends and popleft operations.

Complete the method that removes the next URL.

from collections import deque
queue = deque(["/1", "/2"])
print(queue.____())

Answer: popleft

Why: popleft removes the leftmost item from a deque.

Complete the set insertion call.

seen = set()
url = "/a"
seen.____(url)
print(len(seen))

Answer: add

Why: Set.add records a URL as visited.

Complete the membership check.

seen = {"/a"}
url = "/a"
print(url in ____)

Answer: seen

Why: Checking membership in the visited set prevents a second visit.

What does the program output?

links = ["/a", "/b", "/a"]
print(list(dict.fromkeys(links)))

Answer: ['/a', '/b']

Why: dict.fromkeys preserves first-seen order while removing duplicates.

What appears on the screen?

pages = {"/1": ["/2"], "/2": ["/3"], "/3": []}
queue = ["/1"]
seen = []
while queue:
    url = queue.pop(0)
    if url in seen:
        continue
    seen.append(url)
    queue.extend(pages.get(url, []))
print(seen)

Answer: ['/1', '/2', '/3']

Why: The queue visits each page once in breadth-first order.

What is printed?

seen = set()
for url in ["/a", "/a", "/b"]:
    seen.add(url)
print(len(seen))

Answer: 2

Why: The set contains two unique URLs.

What is the output?

from urllib.parse import urlsplit, urlunsplit
parts = urlsplit("https://example.com/guide.html#top")
print(urlunsplit((parts.scheme, parts.netloc, parts.path, parts.query, "")))

Answer: https://example.com/guide.html

Why: Replacing the fragment with an empty string removes #top.