Per-row token coverage (batch lane only; null on serial B=1 caches,
where offset IS the row's count). mlx-lm's ArraysCache tracks no
offset at all (cache.py:594-722) — per-sequence coverage is the
server's bookkeeping; ours lives here because prompt-cache put()
keys on EXACT coverage (#putOrDispose offset === tokens.length) and
merged rows carry different totals. Seeded by mergeRows, advanced in
lockstep by advance(), row-filtered by filter().
Live speculative verify round; null outside a verify transaction.
Methods may retain different verified prefixes for each active request.
Physical cache storage consumed by one additional token for one row. Recurrent caches return 0 because their state does not grow with the sequence. Optional for stateless/training adapters that are never admitted or persisted.
Recurrent state cannot be trimmed: only a row whose own advance count is the key.
Row i as a fresh SERIAL cache — port of mlx-lm ArraysCache.extract
(cache.py:673-676: cache.cache = [c[idx : idx + 1] for c in self.cache],
a B-axis slice of every state slot). A proper subset gets compact storage
so retaining one row does not pin the other rows. A singleton retains an
independent handle over its complete allocation. Offset =
the row's OWN coverage, not the shared max.
Evict rows not in keep (ascending row indices), in place.
Can trim(n) drop the last n tokens? (Ring caches lose trimability
once wrapped.)
Decode: one new position per row (S = 1), with no prefill padding in
effect.
Window: S positions per row, for prefill chunks and tails, speculative
verify windows and committed spans. Padding prepared through
PaddedPrefillCache holds: padded positions change neither state, each
row's convolution tail ends at its last real position, and each row's offset
advances by its real count. Outputs at padded positions are unspecified.
Row i's own token coverage (per-row when batched, offset serial).
Stable storage identity for compatibility guards and persistence. REQUIRED: every capability predicate (isPlainKvCache, isRotating*, packages/inference/src/state/persistence.ts codecs) is a string compare on this value, so a missing override used to fail OPEN into wrong routing (PRs #42/#43: BatchedRotatingCache shipped without one and batched joins dropped running rows). Wrappers report their own kind plus the inner kind.
Speculative verify-round support for NON-trimmable recurrent caches
(SSMCache — gated-DeltaNet conv + recurrent state). A cache exposing
these is spec-eligible without isTrimmable(): the serve loop arms a
round before the verify forward (the layer snapshots pre-round state —
free, MLX arrays are immutable — and records its position-local kernel
inputs), then resolves it: commit on full accept, or rollback(keep)
which restores the snapshot and bit-exactly REPLAYS the first keep
window tokens through the recurrence (identical arithmetic prefix ⇒
identical state). Trimmable caches leave these unset and keep trim().
Drop the last n tokens (future writes overwrite them). bypass (default
false → unchanged behavior) skips the isTrimmable guard for the
speculative-decode rollback case (trim the last n ≤ γ tips right after a
1 concat write); rotating caches then physically shrink the buffer tail.
StaticmergeMerge a fully-prefilled solo row into a running batched cache (prev
null ⇒ the solo row becomes the batch). The returned cache owns fresh
arrays — or, when there is nothing to concat, arrays STOLEN from solo
(nulled there so the scheduler's unconditional dispose of the solo
caches stays safe). prev/solo disposal remains the caller's job.
Recurrent storage behind the same row-layout port as attention KV. Inputs to merge are borrowed; request membership is owned by the caller.