Lex the arguments of ONE statement as a TOKEN STREAM, skipping what it cannot
decode (#316).
Same text, same from, same boundary rules and the same reason strings as
lexArguments — the difference is only what a refusal does. Where the
strict reading discards the whole list, this one publishes the token it could
not decode with undecided set, no value, and its bytes located, then
carries on to the next token. It is the reading every TOKEN-LEVEL rule wants
(findMissingSeparators, the arg fill), and the one no RENDERING consumer
may have: a token with undecided set has no value to send.
Skip-and-continue, not stop-at-refusal
The cheaper design — publish the prefix decoded before the first refusal,
which is what lexValueAnchors does for values — was measured and
rejected. On the 948-script pinned corpus the prefix reading recovers 3,841
argument tokens and finds zero additional missing-separator runs, because
a refusal is usually the FIRST interesting token in the statement
(… interface=$x /ip/route add … has an empty prefix). Skipping past it is
what reaches the rest of the operand stream, and the corpus numbers for both
designs are in commands/explain/README.md → The token stream a rule reads.
Where it still stops
Two constructs have no knowable end, so neither walk can resume past them: an
unterminated string and an unbalanced (/[/{. A ; is a third, and
deliberately so — it is a statement boundary the segmenter owns, and resuming
after it would read the NEXT statement's bytes as this one's arguments.
Lex the arguments of ONE statement as a TOKEN STREAM, skipping what it cannot decode (#316).
Same text, same
from, same boundary rules and the same reason strings as lexArguments — the difference is only what a refusal does. Where the strict reading discards the whole list, this one publishes the token it could not decode withundecidedset, novalue, and its bytes located, then carries on to the next token. It is the reading every TOKEN-LEVEL rule wants (findMissingSeparators, theargfill), and the one no RENDERING consumer may have: a token withundecidedset has no value to send.Skip-and-continue, not stop-at-refusal
The cheaper design — publish the prefix decoded before the first refusal, which is what lexValueAnchors does for values — was measured and rejected. On the 948-script pinned corpus the prefix reading recovers 3,841 argument tokens and finds zero additional missing-separator runs, because a refusal is usually the FIRST interesting token in the statement (
… interface=$x /ip/route add …has an empty prefix). Skipping past it is what reaches the rest of the operand stream, and the corpus numbers for both designs are incommands/explain/README.md→ The token stream a rule reads.Where it still stops
Two constructs have no knowable end, so neither walk can resume past them: an unterminated string and an unbalanced
(/[/{. A;is a third, and deliberately so — it is a statement boundary the segmenter owns, and resuming after it would read the NEXT statement's bytes as this one's arguments.