Python URL parsing

Scraped pages are full of relative links, tracking parameters, and fragments. Treating them as plain strings creates duplicates and broken requests. urllib.parse gives you a small, standard toolkit for splitting, joining, and rebuilding URLs. You will also normalize URLs for deduplication and reject unsafe schemes or hosts.

Split a URL into components

urlparse separates the scheme, network location, path, query, and fragment. The fragment is for the browser and is not sent to the server. Reading components separately is safer than slicing the string at the first slash.

from urllib.parse import urlparse

url = urlparse("https://example.com/docs/page.html?q=python#intro")
print(url.scheme)
print(url.netloc)
print(url.path)
print(url.query)

Five parts, one object

Resolve relative links against the page

urljoin resolves a relative reference against the page URL. It handles ../, root-relative paths, and query-only links. Always join before deduplicating; /docs and https://example.com/docs are the same destination but different strings.

from urllib.parse import urljoin

base = "https://example.com/docs/page.html"
print(urljoin(base, "../guide.html"))
print(urljoin(base, "/about"))
print(urljoin(base, "?page=2"))

One base, three kinds of link

Read query parameters

parse_qs turns a query string into a dictionary of lists because a key can appear more than once. Use parse_qsl when order matters. Values are still strings, so convert page numbers explicitly before using them in arithmetic.

from urllib.parse import parse_qs

params = parse_qs("q=python&page=2&tag=a&tag=b")
print(params["page"])
print(params["tag"])

Repeated keys become lists

Build a query string safely

urlencode escapes spaces, ampersands, and non-ASCII text for you. Pass a mapping or a sequence of pairs. Do not concatenate raw user input into a URL; one stray ampersand can change the meaning of the request.

from urllib.parse import urlencode

query = urlencode({"q": "a b", "page": 2})
print(query)

Encode instead of concatenating

Normalize before deduplication

The same page can be written with a different case, default port, fragment, or query order. Normalize scheme and host to lowercase, remove the default port, sort the query, and drop the fragment before comparing URLs. Keep the path case-sensitive.

from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit

def normalize(url):
    parts = urlsplit(url)
    scheme = parts.scheme.lower()
    host = (parts.hostname or "").lower()
    default_port = 443 if scheme == "https" else 80
    port = parts.port
    netloc = host if not port or port == default_port else f"{host}:{port}"
    query = urlencode(sorted(parse_qsl(parts.query, keep_blank_values=True)))
    return urlunsplit((scheme, netloc, parts.path or "/", query, ""))

for raw in [
    "HTTPS://Example.com:443/docs?b=2&a=1#top",
    "https://example.com/docs?a=1&b=2",
]:
    print(normalize(raw))

Two spellings, one key

Reject unsafe schemes and unexpected hosts

urljoin follows whatever scheme the link provides, including javascript: and data:. Parse the result and allow only http or https plus an expected host before fetching it. A relative link can still become an external URL after joining, so check after resolution as well as before.

from urllib.parse import urljoin, urlsplit

base = "https://example.com/docs/page"
for link in ["/next", "javascript:alert(1)", "https://evil.example/x", "//example.com/other"]:
    target = urljoin(base, link)
    parts = urlsplit(target)
    safe = parts.scheme in {"http", "https"} and parts.hostname == "example.com"
    print(link, safe, target)

Resolve first, then allowlist

Try it yourself

Change the values, predict the output, then run the code. The Python runtime is downloaded the first time you press Run, so the first run can take a few seconds.