How XPath works
XPath treats an XML document or parsed HTML page as a tree of nodes. An expression describes a path through that tree. It can select nodes by element name, attribute, position, text or relationships such as parent and sibling.
HTML: <article data-id="42"><h2><a>Blue Widget</a></h2></article>
XPath: //article[@data-id="42"]/h2/a/text()
Result: Blue WidgetHere, //article finds article elements anywhere in the document. [@data-id="42"] filters by attribute and text() selects the link’s text node.
Using XPath in web scraping
A web scraper first downloads or renders a page. Its parser then evaluates XPath expressions against the resulting document tree and passes the selected values to your data parsing code.
XPath doesn’t fetch pages, execute scripts or send network traffic. Configure any proxy in your HTTP client, crawler or browser before loading the page. If content appears only after JavaScript rendering, evaluate XPath against a rendered browser DOM rather than the initial HTML response.
XPath vs CSS selectors
Both methods select content from HTML, but their navigation features differ.
| Feature | XPath | CSS selector |
|---|---|---|
| Basic selection | Elements and attribute nodes | Elements by tag, class, ID or attribute |
| Navigation | Down, up and across the tree | Mainly descendants and siblings |
| Text matching | Supports text predicates | Not generally supported |
| Best fit | Structural or text-based queries | Class, ID and attribute queries |
A CSS selector is usually easier to read for simple targets. XPath is useful when you must find an element through its text, parent or ancestor.
Writing reliable XPath
- Prefer stable attributes. Target
id,data-*or meaningful class values rather than visual position. - Avoid long absolute paths. Expressions such as
/html/body/div[2]/div[3]break when a wrapper is added. - Scope the query. Start from a product card or table row before selecting its price or link.
- Handle missing and duplicate nodes. Test whether an expression returns zero, one or many results.
- Check implementation support. Browsers commonly support XPath 1.0, while XML tools may support newer versions.
FAQ
Can XPath be used with HTML?
Yes. Browsers and scraping libraries parse HTML into a document tree that XPath can query, even though XPath was designed for XML data models.
Why does XPath work in my browser but not in my scraper?
The browser may have rendered JavaScript or repaired invalid HTML differently. Compare the scraper’s parsed document with the live DOM and check whether the target is inside an iframe.
Can XPath send requests through a proxy?
No. XPath only selects nodes from an existing document. Set the proxy in the HTTP client or browser that loads the page, then run XPath on the returned content.