XPath for Web Scraping and Data Extraction

Test automation uses XPath to find one element and act on it. Scraping uses it to pull values out of many elements at once. The expressions lean on text(), @attr, string() and normalize-space() far more, and you get to use expressions that return strings and numbers, which locators cannot.

The Python toolchain

lxml is the engine (built on libxml2, XPath 1.0 plus EXSLT). parsel wraps it with a nicer API and is what Scrapy uses. BeautifulSoup does not support XPath; pair it with lxml if you want both.

from lxml import html

tree = html.fromstring(page_source)
titles = tree.xpath("//h2[@class='title']/text()")
links  = tree.xpath("//a[contains(@class, 'result')]/@href")
price  = tree.xpath("string(//span[@data-testid='price'])")

from parsel import Selector

sel = Selector(text=page_source)
sel.xpath("//h2[@class='title']/text()").getall()
sel.xpath("//a[contains(@class, 'result')]/@href").get()
sel.xpath("normalize-space(//span[@data-testid='price'])").get()

# Scrapy spider
def parse(self, response):
    for card in response.xpath("//article[@data-testid='product-card']"):
        yield {
            "name":  card.xpath("normalize-space(.//h3)").get(),
            "price": card.xpath(".//span[@class='price']/text()").get(),
            "url":   card.xpath(".//a/@href").get(),
        }

The container-then-relative pattern (card.xpath(".//h3")) is the same one used for lists and repeated components in tests. Note the . before //.

Extracting text

Expression	Returns
//h2/text()	list of the h2’s direct text nodes
//h2//text()	every text node under the h2, including nested elements
string(//h2)	one string: the first h2’s full text
normalize-space(//h2)	the same, trimmed and collapsed
//h2/text()[1]	first text node only
//p/text()[normalize-space()]	text nodes that are not just whitespace

For “all the visible text of this element as one string”, use normalize-space() on the element, or join .//text() in Python. For a list of paragraphs, use //p/text() and clean each one.

Extracting attributes

//a/@href                          every href
//img/@src | //img/@data-src       lazy-loaded images use data-src
//meta[@property='og:title']/@content
//*[@data-price]/@data-price
//input[@name='csrf_token']/@value

Following structure

Scraped pages rarely have test ids, so axes matter more:

//dt[normalize-space()='Weight']/following-sibling::dd[1]       definition lists
//th[normalize-space()='Price']/following-sibling::td[1]        key-value table rows
//h2[normalize-space()='Specifications']/following-sibling::ul[1]/li
//tr[td[1][normalize-space()='Total']]/td[2]
//table[.//th[normalize-space()='Ticker']]//tr[position() > 1]  the table with that header

Computing in XPath

Because you are not restricted to node-set results, aggregate directly:

count(//article[@data-testid='product-card'])
sum(//td[@class='qty'])
sum(//span[@class='price']/translate(., '$,', ''))    NOT valid: functions cannot be in a path step

The last line is a reminder: in XPath 1.0 you cannot apply a function per node inside a path. Get the list with XPath and do the arithmetic in Python.

Regex with EXSLT

libxml2 (and therefore lxml, parsel, Scrapy) supports the EXSLT regular expression extension. Register the namespace and you get re:test, re:match and re:replace:

ns = {"re": "http://exslt.org/regular-expressions"}
tree.xpath("//a[re:test(@href, '/product/\\d+$')]/@href", namespaces=ns)

parsel adds a convenience: sel.xpath("//h2/text()").re(r"Chapter (\d+)").

This is a libxml2 feature. It does not exist in browsers or Selenium.

XML feeds and namespaces

Sitemaps, RSS, Atom and SOAP use default namespaces, so unprefixed names fail. Either use local-name() or register a prefix:

ns = {"s": "http://www.sitemaps.org/schemas/sitemap/0.9"}
tree.xpath("//s:url/s:loc/text()", namespaces=ns)
tree.xpath("//*[local-name()='loc']/text()")      # portable, no registration

Dynamic pages

lxml parses the HTML the server sent. If the content is rendered by JavaScript, the elements are not in that HTML. Options: find the JSON API the page calls (often simpler), or drive a browser with Playwright and evaluate XPath there:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    page = p.chromium.new_page()
    page.goto(url)
    names = page.locator("//article[@data-testid='product-card']//h3").all_inner_texts()

Other languages

• Java: javax.xml.xpath (XPath 1.0) on a DOM, or jsoup for CSS (no XPath). Saxon adds XPath 3.1.
• JavaScript (Node): xpath package with xmldom, or run Playwright/Puppeteer for rendered pages.
• Go: htmlquery / xpath packages (antchfx).
• .NET: HtmlAgilityPack with SelectNodes("//a/@href"), XPath 1.0.
• Command line: xmllint --html --xpath "//h1/text()" page.html, xidel.

Ethics and limits

Check the site’s terms and robots.txt, rate-limit your requests, identify your bot in the user agent, and cache aggressively. Scraping is legal in many contexts and prohibited in others; XPath knowledge does not change that.

Next Steps

1. Text Matching Functions - The extraction toolkit
2. SVG, Namespaces and local-name() - Namespaced XML in depth
3. XPath 1.0 vs 2.0 vs 3.1 - Which processors give you regex and JSON natively