XPath, from scratch

An XPath expression is evaluated against a context node and returns one of exactly four things in XPath 1.0: a node-set, a string, a number or a boolean. Most expressions you write are location paths, which return node-sets, and which read like filesystem paths on purpose.

<catalog>
  <book id="b1" lang="en">
    <title>Learning XML</title>
    <author>Erik Ray</author>
    <price currency="GBP">29.99</price>
  </book>
  <book id="b2" lang="en">
    <title>XML in a Nutshell</title>
    <author>Elliotte Harold</author>
    <author>W. Scott Means</author>
    <price currency="GBP">34.50</price>
  </book>
  <book id="b3" lang="de">
    <title>XML kompakt</title>
    <price currency="EUR">18.00</price>
  </book>
</catalog>

A leading / makes the path absolute and starts at the document node, which is the invisible parent of the root element, not the root element itself. That is why /catalog works and /catalog/catalog does not. A leading // starts anywhere: it means "search the whole document at any depth". No leading slash makes the path relative to whatever context node the host language gave you.

Every step in a path has the same three-part shape: an axis, a node test, and zero or more predicates. The paths above use abbreviated syntax, where the axis is left implicit. Written out in full, /catalog/book is /child::catalog/child::book, and @id is attribute::id.

XPath 1.0 defines thirteen axes. Five of them have abbreviations, which is why most XPath you read looks nothing like the grammar. The other eight have to be written out, and they are where the language stops being a path syntax and starts being useful.

The node test after the axis is a name, a wildcard, or one of the type tests node(), text(), comment() and processing-instruction(). A wildcard is * for any element on the default axis, or @* for any attribute. Note that text() and node() are node tests rather than functions, despite the parentheses, which is why you cannot pass anything to them.

//title[. = 'XML in a Nutshell']/../@id
      -> b2   (climb from a title to its book's id)

//book[@id='b1']/following-sibling::book/title
      -> XML in a Nutshell, XML kompakt

//author[1]/ancestor::catalog
      -> the catalog element

//book[title='XML kompakt']/preceding-sibling::book
      -> the b1 and b2 elements

count(//book/descendant::text())
      -> the number of text nodes under all books

A predicate is a boolean expression in square brackets that filters the nodes a step selected. Positions are 1-based, not 0-based, so //book[1] is the first book and //book[0] is always empty. A predicate that evaluates to a number is shorthand for position() equals that number, which is why [1] works at all.

Here is the trap. A predicate binds to the step it is attached to, not to the whole path, and it is evaluated once per context node rather than once for the path. So //book[1] does not mean "the first book in the document". It means "every book that is the first book among its own parent's book children", which in a document with several catalogs returns several nodes.

//book[1]

Reads as: for each parent in the document,
take its first book child. Two catalogs
means two results.

Same shape, same problem:
  //div[last()]
  /catalog/book/author[1]

(//book)[1]

The parentheses make the path produce a
node-set first, and the predicate then
filters that single set by position.

Same fix:
  (//div)[last()]
  (/catalog/book/author)[1]

Predicates chain, and each one filters the output of the one before it, so [@lang="en"][2] is the second English book while [2][@lang="en"] is the second book, kept only if it is English. Those are different queries and both are legal. position() and last() are relative to the node-set the current step produced, so //book[position() = last()] is the last book under each parent.

XPath 1.0 has 27 built-in functions and you will use about ten of them. The one distinction worth getting right early is between text() and the string-value of a node. text() selects the immediate text node children of an element. The string-value, which is what you get when a node is used where a string is expected, concatenates every descendant text node. For mixed content those two differ, and the bug looks like missing text.

Two functions people expect and do not get: XPath 1.0 has no ends-with() and no lower-case(). The first is spelled substring(@href, string-length(@href) - 3) = ".xml", and the second is spelled with translate(), which maps characters one to one: translate(@lang, "ABCDEFGHIJKLMNOPQRSTUVWXYZ", "abcdefghijklmnopqrstuvwxyz"). Both arrived properly in XPath 2.0.

Change the sample document's root to <catalog xmlns="urn:books"> and every expression above stops matching. XPath 1.0 has no concept of a default namespace: an unprefixed name in an expression means "in no namespace", so //book cannot reach {urn:books}book. You have to bind a prefix in the host language, and the prefix you bind has nothing to do with any prefix in the document. It is a local variable for a URI.

const doc = new DOMParser().parseFromString(src, 'application/xml');
if (doc.querySelector('parsererror')) throw new Error('not well-formed');

const NS = { bk: 'urn:books', dc: 'http://purl.org/dc/elements/1.1/' };
const resolver = (prefix) => NS[prefix] ?? null;

// Snapshot, not iterator: iterator result types are invalidated by any
// mutation of the document during iteration. Snapshots are not.
const snap = doc.evaluate(
  '//bk:book[@id="b2"]/bk:title',
  doc,
  resolver,
  XPathResult.ORDERED_NODE_SNAPSHOT_TYPE,
  null,
);
for (let i = 0; i < snap.snapshotLength; i++) {
  console.log(snap.snapshotItem(i).textContent);
}

// An unbound prefix throws a DOMException with legacy code 14,
// NAMESPACE_ERR. A malformed expression throws a different one.
// Both need catching separately from an empty result.

from lxml import etree

tree = etree.fromstring(src.encode('utf-8'))
NS = {'bk': 'urn:books', 'dc': 'http://purl.org/dc/elements/1.1/'}

for title in tree.xpath('//bk:book[@id="b2"]/bk:title', namespaces=NS):
    print(title.text)

# lxml refuses an empty prefix outright:
#   tree.xpath('//book', namespaces={'': 'urn:books'})
#   ValueError: empty namespace prefix is not supported in XPath
# Bind a made-up prefix instead. It never appears in the document.

# The standard library's ElementTree understands a subset of XPath and
# the same namespaces dict, plus Clark notation in element names:
#   root.findall('{urn:books}book/{urn:books}title')

DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
factory.setNamespaceAware(true);          // defaults to FALSE
factory.setFeature(javax.xml.XMLConstants.FEATURE_SECURE_PROCESSING, true);
Document doc = factory.newDocumentBuilder().parse(input);

XPath xpath = XPathFactory.newInstance().newXPath();
xpath.setNamespaceContext(new NamespaceContext() {
    public String getNamespaceURI(String prefix) {
        return "bk".equals(prefix) ? "urn:books" : XMLConstants.NULL_NS_URI;
    }
    public String getPrefix(String uri) { return null; }
    public java.util.Iterator<String> getPrefixes(String uri) { return null; }
});

NodeList nodes = (NodeList) xpath.evaluate(
    "//bk:book[@id='b2']/bk:title", doc, XPathConstants.NODESET);
for (int i = 0; i < nodes.getLength(); i++) {
    System.out.println(nodes.item(i).getTextContent());
}

If registering a prefix is genuinely awkward, there is one portable escape hatch: //*[namespace-uri()="urn:books" and local-name()="book"]. It works in every engine and needs no resolver. It is also unreadable at any length, so treat it as a debugging tool rather than a style. The XPath tester on this site collects the declarations from your document automatically and builds the resolver for you, and warns when a document has a default namespace and your expression carries no prefix, because the symptom otherwise is an empty result that looks exactly like missing data.

2.0 is the real break. It replaces the four-type model with XDM, the XQuery and XPath Data Model, in which every value is a sequence of items and an item is a node or a typed atomic value. It adds for, if and quantified expressions, instance of and cast as, the range operator to, the set operators intersect and except, node identity comparison with is, and regular expressions through matches(), replace() and tokenize(). Appendix I of the Recommendation lists what this breaks: with compatibility mode off, comparing a sequence of more than one item to a single value raises an error rather than quietly behaving existentially the way 1.0 does.

3.0 adds function items, inline functions, let expressions and the string concatenation operator ||. 3.1 adds maps and arrays as first-class item types, the lookup operator ? for reaching into them, the arrow operator => for chaining calls, and JSON handling through parse-json() and xml-to-json().

None of that is in a browser. Every engine implements XPath 1.0 and only XPath 1.0. The API was originally specified in DOM Level 3 XPath, now a retired W3C Working Group Note dated 3 November 2020 and explicitly marked as not to be used for further technical work; the living definition is section 8 of the WHATWG DOM Standard. Firefox additionally does not implement the namespace:: axis, so do not build anything on it. The same ceiling applies to libxml2, and therefore to lxml and to xmllint: XPath 1.0. For 2.0 or later on the server you need Saxon, or elementpath in Python, or a database engine such as BaseX.