A token in the total, gapless byte partition (B1).
Every byte of the analyzed input belongs to exactly one token — a
contiguous, non-overlapping partition of [0, input.bytes) — sorted by
start, with no gaps and tokens.map(t => slice).join("") === input.
Byte-exact, inheriting the #215/#252 offset discipline. Offsets are on the
analyzed text; input.positionMap applies when normalized.
class is explicitly provisional until #264 B5 — the vocabulary here is
not the final LSP/SCIP legend, and every byte no analyzer claims becomes
unclassified. Filling those holes is B2, one PR per fill. unclassified
is a first-class answer, not a placeholder to be avoided.
B2 so far: operator (26 spellings + 2 aliases, syntax-meta is a residual
merge, never a source, #255), arg (argument names and their = as located
by args.ts — the name run is [span.start, valueSpan.start - 1) and the =
is the single byte at valueSpan.start - 1), and value (argument value
bytes and leaf array-literal members from data.values.occurrences — leaves
only, quotes included, #295). arg covers both the name bytes and the =
byte in one provisional class and value covers every leaf value span
(emit first, name later); whether the = or per-shape value later deserve
their own classes is #264 B5 vocabulary and does not move the byte coverage.
A token in the total, gapless byte partition (B1).
Every byte of the analyzed input belongs to exactly one token — a contiguous, non-overlapping partition of
[0, input.bytes)— sorted bystart, with no gaps andtokens.map(t => slice).join("") === input. Byte-exact, inheriting the #215/#252 offset discipline. Offsets are on the analyzed text;input.positionMapapplies when normalized.classis explicitly provisional until #264 B5 — the vocabulary here is not the final LSP/SCIP legend, and every byte no analyzer claims becomesunclassified. Filling those holes is B2, one PR per fill.unclassifiedis a first-class answer, not a placeholder to be avoided.B2 so far:
operator(26 spellings + 2 aliases,syntax-metais a residual merge, never a source, #255),arg(argument names and their=as located byargs.ts— the name run is[span.start, valueSpan.start - 1)and the=is the single byte atvalueSpan.start - 1), andvalue(argument value bytes and leaf array-literal members fromdata.values.occurrences— leaves only, quotes included, #295).argcovers both the name bytes and the=byte in one provisional class andvaluecovers every leaf value span (emit first, name later); whether the=or per-shapevaluelater deserve their own classes is #264 B5 vocabulary and does not move the byte coverage.