The supplier's HTML has tables inside tables, and a regex cannot count nesting. A parser walks the tags in order. Which pieces should you keep, and which should you drop?
HTML is nested structure, not a flat string. Python's html.parser walks opening tags, text, and closing tags in order. Subclass HTMLParser and store only the pieces you need; the result is testable and does not depend on a particular page layout. You will also pair nested tags with a stack and keep script text out of visible output.
Subclass the parser
Create a class that inherits HTMLParser and call super().__init__ before adding your own state. Override handle_starttag to react to opening tags. Then create an instance and feed it an HTML string.
from html.parser import HTMLParser
class Tags(HTMLParser):
def __init__(self):
super().__init__()
self.names = []
def handle_starttag(self, tag, attrs):
self.names.append(tag)
parser = Tags()
parser.feed("<h1>Hello</h1><p>World</p>")
print(parser.names)Opening tags in order
Turn attributes into a dictionary
The attrs argument is a list of name-value pairs. Convert it with dict when you need a lookup by name. Check for None: boolean attributes can appear without a value, and a missing href should not become a broken link.
from html.parser import HTMLParser
class Links(HTMLParser):
def __init__(self):
super().__init__()
self.links = []
def handle_starttag(self, tag, attrs):
values = dict(attrs)
if tag == "a" and values.get("href"):
self.links.append(values["href"])
parser = Links()
parser.feed('<a href="/a">A</a><a href="/b">B</a>')
print(parser.links)Read the href safely
Collect visible text
handle_data receives text between tags. Strip whitespace and ignore empty strings so the output does not fill up with indentation. If you need the text of one element only, track whether you are inside that element with a boolean flag.
from html.parser import HTMLParser
class Text(HTMLParser):
def __init__(self):
super().__init__()
self.parts = []
def handle_data(self, data):
text = data.strip()
if text:
self.parts.append(text)
parser = Text()
parser.feed("<h1>Title</h1><p>First sentence.</p>")
print(" | ".join(parser.parts))Text without indentation
Decode entities and keep the source
HTMLParser decodes character references such as & by default. If you store text for later, keep the original URL beside it so the extracted value can be traced back. A parser gives structure; it does not decide whether the page is trustworthy.
from html import unescape
raw = "Tom & Jerry"
print(unescape(raw))
print(raw)Decode without losing the source
Use a stack to pair nested tags
HTMLParser reports start and end tags separately, so a simple flag can lose the matching end tag when elements nest. A stack records the open element's data and pops it on the matching close. This small parser is fine for controlled fixtures; a full HTML parser is safer for arbitrary pages.
from html.parser import HTMLParser
class LinkCollector(HTMLParser):
def __init__(self):
super().__init__()
self.stack = []
self.links = []
def handle_starttag(self, tag, attrs):
if tag == "a":
self.stack.append(dict(attrs).get("href"))
def handle_endtag(self, tag):
if tag == "a" and self.stack:
href = self.stack.pop()
if href:
self.links.append(href)
parser = LinkCollector()
parser.feed('<a href="/one">One</a><a href="/two"><b>Two</b></a>')
print(parser.links)Push on start, pop on close
Keep script and style text out of visible output
handle_data also receives the contents of script and style elements. Count those elements while parsing and ignore their text. Otherwise a JavaScript string can appear in a summary that was supposed to contain only visible page text.
from html.parser import HTMLParser
class VisibleText(HTMLParser):
def __init__(self):
super().__init__()
self.skip = 0
self.parts = []
def handle_starttag(self, tag, attrs):
if tag in {"script", "style"}:
self.skip += 1
def handle_endtag(self, tag):
if tag in {"script", "style"} and self.skip:
self.skip -= 1
def handle_data(self, data):
if not self.skip and data.strip():
self.parts.append(data.strip())
parser = VisibleText()
parser.feed("<h1>Menu</h1><script>alert('x')</script><p>Tea</p>")
print(" | ".join(parser.parts))Visible text only
Key points
- Subclass HTMLParser and call super().__init__ before storing state.
- handle_starttag receives the tag name and a list of attribute pairs.
- handle_data receives text nodes, including whitespace.
- Use feed to process an HTML string; the parser calls your handlers.
- Keep the source URL with extracted text so results can be checked.
- Use a stack for paired elements; skip script/style text when extracting visible content.
Common mistakes
- Forgetting super().__init__ and losing the parser's internal state.
- Assuming attrs is already a dictionary.
- Using a regex to parse nested HTML instead of using a parser.
- Using one boolean for nesting and attaching text to the wrong closing tag.
Extract every link as a (label, href) tuple from a saved HTML string, skipping links without an href.
Try it yourself
Change the values, predict the output, then run the code. The Python runtime is downloaded the first time you press Run, so the first run can take a few seconds.
Practice this lesson
12 questions with immediate explanations.