The per-forward masks of the model this cache belongs to. Row layouts
made from this cache (state/layout) share them.
ReadonlymasksThe per-forward masks of the model this cache belongs to. Row layouts
made from this cache (state/layout) share them.
ReadonlymaxOptionalminimumEarliest prefix whose retained representation can resume generation. Irreversible transitions may preserve bytes while invalidating older precision boundaries. Reuse policy must not trim below this offset.
Optional ReadonlyropeSet only by compiled-decode trace adapters (see Cache).
Static ReadonlySTEPGemma 4's vision prefill read on a sliding layer: bidirMask with this
cache's window over a first window, then the fused SDPA.
A committed span, one decode per position (decodeByPosition): the
in-place ring write and read each position would get.
Decode: one new position per row (L = 1). Each row's query sees that
row's valid stored positions up to and including the one just appended,
limited to the cache's sliding window.
DiffusionGemma's vision encoder read on a sliding layer, as its encoder
computes it: the window's concatenating append (every position kept, even
past maxSize), then the fused SDPA under the encoder window's mask
(encoderWindowMask), which this cache's window does not limit.
Window: L new positions per row, causal within the window, for prefill
chunks, prefill tails and speculative verify windows. Query i of a row
sees the row's earlier valid positions and window positions 0..i, limited
to the cache's sliding window. Left padding and prepared prefill padding are
never visible. Outputs at padded query positions are unspecified and the
caller ignores them. To append context without attending, make this call
and dispose the read unused.
Physical cache storage consumed by one additional token for one row. Recurrent caches return 0 because their state does not grow with the sequence. Optional for stateless/training adapters that are never admitted or persisted.
The live window as donor rows (see temporalView).
Apply the closure's outputs (takes ownership); returns arrays to evaluate with the step.
The closure inputs of slot, in order.
Non-mutating twin of prepareDecodeStep's fetch choice: the ring runs once the next write lands at or past the window.
Port of mlx-lm RotatingKVCache.is_trimmable/trim: only valid while the ring has never wrapped (still in temporal order).
Compiled decode: host-side bookkeeping of #updateInPlace (L=1) — growth, oversize trim, rotation — without the write.
DiffusionGemma's canvas read on a sliding layer, as its decoder computes
it: the live window in temporal order, cut to the newest maxSize - 1
positions once it is full, then the block's keys and values. The mask is
the decoder's (stored positions past the window start and the whole block),
sliced to the kept columns.
Stable storage identity for compatibility guards and persistence. REQUIRED: every capability predicate (isPlainKvCache, isRotating*, packages/inference/src/state/persistence.ts codecs) is a string compare on this value, so a missing override used to fail OPEN into wrong routing (PRs #42/#43: BatchedRotatingCache shipped without one and batched joins dropped running rows). Wrappers report their own kind plus the inner kind.
The live window in chronological order: the newest min(offset, maxSize) positions. A multi-token write leaves the buffer oversized in temporal order (#updateConcat keeps maxSize - 1 history rows plus the whole block), so the window is the tail of that order, never its head. optiq's kv_view._read_cache_temporal takes the head, which is the oldest rows in that state; the batched layouts and rotatingSourcePosition already take the tail.
Port of optiq rotating.py _replay_into_quantized: quantize the
whole buffer AS-LAID-OUT (ring order, not temporal order — correct
because ringIdx is preserved with it) into a
RotatingQuantizedKVCache, which attends with the kernel set kernels
builds for its bits, group size and this cache's activation dtype.
Drop the last n tokens (future writes overwrite them). bypass (default
false → unchanged behavior) skips the isTrimmable guard for the
speculative-decode rollback case (trim the last n ≤ γ tips right after a
1 concat write); rotating caches then physically shrink the buffer tail.
Rotating (sliding-window) KV cache — port of mlx-lm RotatingKVCache with keep=0 (gemma4's configuration): a ring buffer of max_size entries, so decode attends over at most the window. RoPE offsets use the true position; masks use the buffer-clamped offset.
Reads: the sliding window is
maxSize, the window every graph passes for the layers it gives this cache. Decode writes in place at the ring head and attends the whole ring with no mask (eviction enforces the window). A window writes by concatenation, in temporal order, and attends under the causal mask, materialized with the sliding window once the window binds. Both use the stock fused SDPA, as the graphs do afterupdateAndFetchtoday, which picks the in-place write for one position and concatenation otherwise. The method picks here, so a one-row window concatenates: the same bits until the ring wraps, a different key order after.