The per-forward masks of the model this cache belongs to. Row layouts
made from this cache (state/layout) share them.
The TurboQuant read of the graph that built this cache: a TurboQuant
conversion of this storage (TurboQuantKVCache.fromKVCache, the delayed
TurboQuant rows) attends with it, and copies of this cache carry it.
Qwen 3.5's makeCache passes denseTurboQuantKernels, Gemma 4's and
MiniCPM5's deferredTurboQuantKernels; a graph that reads TurboQuant
storage only through the deprecated fetches passes none.
The read of a causal window (appendWindow, appendCommitted): the
stock fused SDPA, or the read the composition names
(foldedWindowRead, bit for bit the same). Row layouts made from this
storage and its clones read their windows with it.
ReadonlymasksThe per-forward masks of the model this cache belongs to. Row layouts
made from this cache (state/layout) share them.
Optional ReadonlyropeSet only by compiled-decode trace adapters (see Cache).
ReadonlyturboThe TurboQuant read of the graph that built this cache: a TurboQuant
conversion of this storage (TurboQuantKVCache.fromKVCache, the delayed
TurboQuant rows) attends with it, and copies of this cache carry it.
Qwen 3.5's makeCache passes denseTurboQuantKernels, Gemma 4's and
MiniCPM5's deferredTurboQuantKernels; a graph that reads TurboQuant
storage only through the deprecated fetches passes none.
ReadonlywindowThe read of a causal window (appendWindow, appendCommitted): the
stock fused SDPA, or the read the composition names
(foldedWindowRead, bit for bit the same). Row layouts made from this
storage and its clones read their windows with it.
Static ReadonlySTEPGemma 4's vision prefill read: the causal mask OR'd with the image block
overlay (bidirMask), then the fused SDPA with that array mask. One mask
per forward: every layer passes the same bidirectional tensor.
A bf16 committed span reads exactly as a window does.
Decode: one new position per row (L = 1). Each row's query sees that
row's valid stored positions up to and including the one just appended,
limited to the cache's sliding window.
DiffusionGemma's vision encoder read on a full-attention layer: the
append updateAndFetch makes, then the fused SDPA under the encoder
window's mask (encoderWindowMask), one per forward for every layer.
Window: L new positions per row, causal within the window, for prefill
chunks, prefill tails and speculative verify windows. Query i of a row
sees the row's earlier valid positions and window positions 0..i, limited
to the cache's sliding window. Left padding and prepared prefill padding are
never visible. Outputs at padded query positions are unspecified and the
caller ignores them. To append context without attending, make this call
and dispose the read unused.
Physical cache storage consumed by one additional token for one row. Recurrent caches return 0 because their state does not grow with the sequence. Optional for stateless/training adapters that are never admitted or persisted.
Chronological (K, V) view sliced to offset (drafter donor read).
Apply the closure's outputs (takes ownership); returns arrays to evaluate with the step.
The closure inputs of slot, in order.
Non-mutating: the fetch prepareDecodeStep will choose.
Can trim(n) drop the last n tokens? (Ring caches lose trimability
once wrapped.)
Compiled decode: host-side bookkeeping for a 1-token step — the growth/trim half of updateAndFetch (L=1), without the write.
DiffusionGemma's canvas read on a full-attention layer: the stored positions in order, then the block's keys and values, with no mask.
Stable storage identity for compatibility guards and persistence. REQUIRED: every capability predicate (isPlainKvCache, isRotating*, packages/inference/src/state/persistence.ts codecs) is a string compare on this value, so a missing override used to fail OPEN into wrong routing (PRs #42/#43: BatchedRotatingCache shipped without one and batched joins dropped running rows). Wrappers report their own kind plus the inner kind.
Arrays to eval to materialize cache state (prefill chunk boundary).
Port of mlx-lm KVCache.to_quantized: quantize the whole buffer
(padding included — it's overwritten before being read). The converted
cache attends with the kernel set kernels builds for its bits, group
size and this cache's activation dtype.
Drop the last n tokens (future writes overwrite them). bypass (default
false → unchanged behavior) skips the isTrimmable guard for the
speculative-decode rollback case (trim the last n ≤ γ tips right after a
1 concat write); rotating caches then physically shrink the buffer tail.
KV cache — port of mlx-lm cache.py KVCache: preallocated in steps of 256 along the sequence axis, updated in place via slice_update.
Reads: each named read appends through
updateAndFetchand attends the fetched prefix with the stock fused SDPA, the call every graph makes afterupdateAndFetchtoday, a window throughwindowRead(that SDPA unless the composition names a read computing its bits otherwise). Decode passes no mask and a window the fused causal mask; at one query the two compute the same bits, so a one-row window needs no case of its own. This cache has no sliding window (graphs composeRotatingKVCachefor sliding layers), so only the bidirectional and encoder reads build a mask array.