OptionalcacheOptionalcacheOptionalcompiledResolved replay permission; standalone group callers retain auto mode.
OptionalcontextOptionalcontinuationOptional ordinary continuation: its checkpoint key and sampler recovery; the cache service resumes and checkpoints it.
OptionalgrammarGrammar controller for this row (B1: per-row matchers). When set, the
scheduler drives it: accept() after each emitted token (fires the async
bitmask fill), await ready() before the row's next sample (the sample
closure applies applyMask). Termination (grammar satisfied) is an
additional per-row stop source. The gateway OWNS disposal (finally around
submit()); the scheduler uses but never owns. Null on the degrade path.
OptionalonCloses the gateway's queue/admission span when this row leaves pending.
Called per emitted (non-EOS) token, in order. Returning false halts this
row (a decoded-text stop sequence fired) — matches generate()'s onToken
contract. EOS terminates the row WITHOUT an onToken call. Throwing evicts
THIS row only (its submit promise rejects; siblings continue). May be
async; keep it cheap — it runs inline in the step loop.
OptionalplainThe row's sampler is PLAIN GREEDY (temperature 0, no curve) with no
logits processors and no grammar — set by the gateway when true. Lets
the scheduler take the vectorized sampling fast path (ONE
log-softmax+argmax over [B,V] instead of B slice/sample/concat graphs)
when EVERY live row qualifies. Numerically identical per row
(row-independent ops, same per-row shapes); the per-row closure path
is the fallback and the composition's vectorGreedySampling: false.
OptionalprefillRequest policy overrides the execution group's captured prefill default.
OptionalpromptBorrowed prepared input; its owner retains it through preparation.
OptionalreusesFalse for a request whose prefix has no identity (prepared media without one): it takes fresh state and its state is not kept. Rows that differ here do not share a group.
OptionalsignalClient/request lifetime. An aborted pending row is removed before admission; a row in prefill or decode is evicted at the next safe scheduler boundary.
OptionalsnapshotStable cache boundary from the server's template probe (the serial lane's options.snapshotAt): the prompt prefix the NEXT turn's render preserves. The scheduler snapshots a trim-free prompt-cache entry at min(this, promptIds.length-1) during the solo prefill — the oracle invariant (mlx-lm insert_segments) that gives wrapped-ring models a reuse path. Absent = still capped at promptIds.length-1.
OptionaltraceDiagnostic-only request-local trace.
Resolve loaded-state identity under the execution lease, after any queued adapter replacement has finished.