The kernel of each read. Row layouts made from this cache keep them.
The per-forward masks of the model this cache belongs to. Row layouts
made from this cache (state/layout) share them.
ReadonlybitsReadonlygroupReadonlykernelsThe kernel of each read. Row layouts made from this cache keep them.
ReadonlymasksThe per-forward masks of the model this cache belongs to. Row layouts
made from this cache (state/layout) share them.
OptionalminimumEarliest prefix whose retained representation can resume generation. Irreversible transitions may preserve bytes while invalidating older precision boundaries. Reuse policy must not trim below this offset.
Optional ReadonlyropeSet only by compiled-decode trace adapters (see Cache).
Static ReadonlySTEPThe kernel set affine storage restored at this cache's layer reads with
(state/persistence): this cache's own kernels.
Committed span: L decided positions per row, causal within the span.
Each query attends its own causal prefix with the arithmetic of a
one-position read, so the span keeps the single-position reduction order
the graph qualifies per KV format (MlxKnownSpans, read by appendKnown). This replaces the
independentPositions flag. The only caller today runs one row with up to
four positions.
Decode: one new position per row (L = 1). Each row's query sees that
row's valid stored positions up to and including the one just appended,
limited to the cache's sliding window.
Window: L new positions per row, causal within the window, for prefill
chunks, prefill tails and speculative verify windows. Query i of a row
sees the row's earlier valid positions and window positions 0..i, limited
to the cache's sliding window. Left padding and prepared prefill padding are
never visible. Outputs at padded query positions are unspecified and the
caller ignores them. To append context without attending, make this call
and dispose the read unused.
Physical cache storage consumed by one additional token for one row. Recurrent caches return 0 because their state does not grow with the sequence. Optional for stateless/training adapters that are never admitted or persisted.
Chronological (K, V) triples sliced to offset — the quantized twin of KVCache.temporalView (batched merge/extend/filter reads). Caller owns the returned views.
Apply the closure's outputs (takes ownership); returns arrays to evaluate with the step.
The closure inputs of slot, in order.
Non-mutating: the fetch prepareDecodeStep will choose.
Can trim(n) drop the last n tokens? (Ring caches lose trimability
once wrapped.)
Compiled decode: growth half of updateAndFetchQuantized (L=1), without quantize/write (those live in the compiled graph / after).
A block read (BlockAttentionCache.readBlock): the stored positions,
then the block's keys and values quantized as this cache stores them and
not appended, read with no mask by the masked-window kernel, which takes
any query count.
Adopt persisted state (takes ownership of the triples' arrays) — the quantized twin of KVCache.restoreState (packages/inference/src/state/persistence.ts).
Stable storage identity for compatibility guards and persistence. REQUIRED: every capability predicate (isPlainKvCache, isRotating*, packages/inference/src/state/persistence.ts codecs) is a string compare on this value, so a missing override used to fail OPEN into wrong routing (PRs #42/#43: BatchedRotatingCache shipped without one and batched joins dropped running rows). Wrappers report their own kind plus the inner kind.
Drop the last n tokens (future writes overwrite them). bypass (default
false → unchanged behavior) skips the isTrimmable guard for the
speculative-decode rollback case (trim the last n ≤ γ tips right after a
1 concat write); rotating caches then physically shrink the buffer tail.
Quantize incoming k/v and append; returns quantized views to offset.
Quantized KV cache — port of mlx-lm QuantizedKVCache: keys/values stored as (packed u32, scales, biases) triples, quantized along head_dim. Only full-attention layers convert (mlx-lm's rotating-cache quantization is NYI upstream; sliding layers are window-capped anyway).
Built empty, it quantizes from token zero: the first append is read quantized, so the first chunk's numbers differ from today's served path and mlx-lm's, which read it bf16 (
Bf16FirstQuantizedKVCache).Reads: each named read appends through
updateAndFetchQuantizedand attends the fetched prefix with the kernelkernelsfixed for it at construction: decode with no mask, a window under the bottom-right causal matrix. The composition names the set (composedAffineKernels: tiled-causal for--kv-quant configand--fused-sdpa on, unfused otherwise). The committed read isquantizedAppendAttention. This cache has no sliding window (graphs composeRotatingQuantizedKVCachefor sliding layers).