Segment original into top-level statements. Spans are analyzed-byte offsets
(see module header); text is the original substring for each span.
A statement-bearing range may anchor the same original text in its
enclosing document. On ASCII input, structural lookback can reuse that
document's statement index instead of indexing every nested body (#322).
The scan still reads the original bytes, including comments: replacing them
with the mask would lose comment spans and H5 continuation state. Non-ASCII
input, or a body opener skipped by the raw document index (a quote inside an
earlier comment), keeps the standalone analyzed-byte scan.
Segment
originalinto top-level statements. Spans are analyzed-byte offsets (see module header);textis the original substring for each span.A statement-bearing
rangemay anchor the sameoriginaltext in its enclosing document. On ASCII input, structural lookback can reuse that document's statement index instead of indexing every nested body (#322). The scan still reads the original bytes, including comments: replacing them with the mask would lose comment spans and H5 continuation state. Non-ASCII input, or a body opener skipped by the raw document index (a quote inside an earlier comment), keeps the standalone analyzed-byte scan.