Building a Tokenizer for XPath or XQuery

This document describes possible strategies for tokenizing the "XML Path Language (XPath) 2.0" and "XQuery 1.0: An XML Query Language" languages, and is provided as a helpful guide to those who are designing an implementation for these languages, and as background material for the normative EBNF found in the language specifications. In the future this document may be expanded to cover more general parsing strategies.