storage: bf16 or affine storage of this layer to adopt (restored,
cloned or extracted elsewhere); affine storage is re-homed to this lego's
kernels, bf16 storage that already holds start positions converts now.
Positions held in bf16 before the conversion; 0 converts after the first append.
The affine kernel set the converted storage attends with, named by the
composition (composedAffineKernels, or the KV4 set).
The per-forward masks of the model this cache belongs to.
Optionalstorage: CacheReadonlybitsReadonlycomposedThe storage the row layouts, clones and saved files take (see the class).
ReadonlygroupReadonlykernelThe affine kernel set the converted storage attends with, named by the
composition (composedAffineKernels, or the KV4 set).
ReadonlymasksThe per-forward masks of the model this cache belongs to.
ReadonlystartPositions held in bf16 before the conversion; 0 converts after the first append.
The donor reads of the storage held now: a drafter borrowing this layer's keys and values reads them as the storage hands them out.
Earliest prefix whose retained representation can resume generation. Irreversible transitions may preserve bytes while invalidating older precision boundaries. Reuse policy must not trim below this offset.
Earliest prefix whose retained representation can resume generation. Irreversible transitions may preserve bytes while invalidating older precision boundaries. Reuse policy must not trim below this offset.
Bidirectional window: the same as appendWindow, except that two
positions both flagged in bidirectional see each other in either
direction, regardless of order or the sliding window. bidirectional is a
borrowed bool [L] that applies to every row. Defined only for the first
window of a sequence (offset 0).
Committed span: L decided positions per row, causal within the span.
Each query attends its own causal prefix with the arithmetic of a
one-position read, so the span keeps the single-position reduction order
the graph qualifies per KV format (MlxKnownSpans, read by appendKnown). This replaces the
independentPositions flag. The only caller today runs one row with up to
four positions.
Decode: one new position per row (L = 1). Each row's query sees that
row's valid stored positions up to and including the one just appended,
limited to the cache's sliding window.
The encoder window, a first window: bf16 storage's read, then the transition its append owes.
Window: L new positions per row, causal within the window, for prefill
chunks, prefill tails and speculative verify windows. Query i of a row
sees the row's earlier valid positions and window positions 0..i, limited
to the cache's sliding window. Left padding and prepared prefill padding are
never visible. Outputs at padded query positions are unspecified and the
caller ignores them. To append context without attending, make this call
and dispose the read unused.
Physical cache storage consumed by one additional token for one row. Recurrent caches return 0 because their state does not grow with the sequence. Optional for stateless/training adapters that are never admitted or persisted.
Apply the closure's outputs (takes ownership); returns arrays to evaluate with the step.
The closure inputs of slot, in order.
Non-mutating: the fetch prepareDecodeStep will choose.
Releases the storage and returns to the constructed state: empty bf16.
Can trim(n) drop the last n tokens? (Ring caches lose trimability
once wrapped.)
Positions an append may add before the transition: up to the start.
Host-side bookkeeping for a 1-token step (growth, rotation), without the write.
The block read of the storage held now (it appends nothing, so nothing converts).
Stable storage identity for compatibility guards and persistence. REQUIRED: every capability predicate (isPlainKvCache, isRotating*, packages/inference/src/state/persistence.ts codecs) is a string compare on this value, so a missing override used to fail OPEN into wrong routing (PRs #42/#43: BatchedRotatingCache shipped without one and batched joins dropped running rows). Wrappers report their own kind plus the inner kind.
Drop the last n tokens (future writes overwrite them). bypass (default
false → unchanged behavior) skips the isTrimmable guard for the
speculative-decode rollback case (trim the last n ≤ γ tips right after a
1 concat write); rotating caches then physically shrink the buffer tail.
Affine-quantized keys and values for one full-attention layer, bf16 first: the lego
--kv-quant N(and akv_config.jsonlayer) composes.Numbers: today's served path and mlx-lm's (
quantized_kv_start). Storage is aKVCacheuntil an append leaves it holding at leaststartpositions (the first append whenstartis 0). That append is read bf16 and then, before it returns, the storage converts (KVCache.toQuantized, the reuse floor at a nonzero start, materialized at once). Every later append quantizes its positions and is read by the quantized storage's own reads, withkernelSet's kernels. Stored values below the offset therefore equal the convert-after path's bit for bit, and so does every read. AQuantizedKVCachebuilt empty is the other composition: it quantizes the first append too, so its first chunk's output differs (tests/state/bf16-first-kv.test.tsrecords by how much), and it has no mlx-lm oracle.The transition keys only on this cache's stored length and runs only inside its appends (or when it adopts storage that already holds
startpositions), so the caller's append boundaries are part of the numbers: a caller that splits one planned prefill chunk into two appends moves the conversion to the split. The rows layout the scheduler batches (DelayedQuantizedKVCache) converts with this same code but only at each row's planned chunk boundary (TransitioningKvRows). The transition is irreversible: trimming below the start afterwards keeps the quantized storage, so a verify window that crosses the start converts even if its tail is later rejected.The row layouts, clones and saved files take the storage, never this cache (
composed): its signature is its own (kv:bf16-first:and the storage's), and its storage is what the convert-after path saves.