Optional ReadonlybareReadonlybosOptional ReadonlyencodeEncode-path counters (memo hits / incremental splices / fallbacks / full encodes) — observability + test hook, no behavioral role.
ReadonlyeosOptional ReadonlytokenizerAbsolute path to tokenizer.json (the model snapshot dir's copy). Set by loadTokenizer; read by packages/inference/src/sampling/grammar.ts to build the xgrammar TokenizerInfo (vocab extraction + vocab-type detection for constrained decoding).
Optional Readonlytrimsmlx-lm streaming-detokenizer parity: true for ByteLevel-decoder (BPE)
tokenizers, where mlx-lm's BPEStreamingDetokenizer drops ONE leading " "
at the very start of a generated sequence (tokenizer_utils.py
_maybe_trim_space: elif not self.text: return current_text[1:]).
Our decode() delegates to @huggingface/tokenizers, whose ByteLevel
decoder KEEPS that space — the serving StreamDecoder consults this flag
to match mlx-lm's rendered bytes (e.g. MiniCPM5 /v1/completions:
"2, 3, 5…" not " 2, 3, 5…"). SPM tokenizers need nothing: with a Strip
decoder the strip already happens inside decode(); without one
(gemma-4) mlx-lm uses trim_space=False and keeps the space too.
Optionalvocabconfig.json vocab_size, may exceed the tokenizer vocab length when the embedding is padded to a power-of-two / 256. Read by packages/inference/src/sampling/grammar.ts to size the logit bitmask against the model's logit width, not the tokenizer's. Set by the server when it loads config; not set by the bare loadTokenizer() helper (callers that need it pass modelDir + config).
OptionalskipSpecialTokens: booleanEncode text → token ids. addSpecialTokens (default true) prepends the BOS;
pass FALSE for chat-template output, which already emits the BOS as text (else
you get a corrupting double-BOS <s><s>…).
OptionaladdSpecialTokens: booleanToken id → raw vocab token string (mlx-lm's convert_ids_to_tokens — the undecoded piece, e.g. "▁Hello", used in logprobs responses). Falls back to decode() for ids outside the base vocab (added tokens).
mlx-lm streaming-detokenizer parity (companion to trimsLeadingSpace): for ByteLevel-decoder (BPE) tokenizers, the vocab id of the bare-space token "Ġ" — the ONLY token matching mlx-lm 0.31.3 BPEStreamingDetokenizer.add_token's hold condition
len(v) == 1 and self._byte_decoder.get(v[0]) == 32(tokenizer_utils.py:206-218, "For single spaces wait until the next token"). mlx_lm.server NEVER calls detokenizer.finalize() (grep: zero hits in server.py 0.31.3), so a generation ENDING on this token silently drops its space from the served bytes. The serving StreamDecoder consults this id to mirror both behaviors (MiniCPM5: id 242).