Positions held in bf16 before the conversion; 0 converts after the first append.
The codec's decode policy for the converted storage.
How the converted storage attends, named by the graph the cache serves
(denseTurboQuantKernels for Qwen 3.5, deferredTurboQuantKernels for
Gemma 4 and MiniCPM5).
The per-forward masks of the model this cache belongs to.
Optionalstorage: CacheReadonlycomposedThe storage the row layouts, clones and saved files take.
ReadonlyfusedThe codec's decode policy for the converted storage.
ReadonlykReadonlymasksThe per-forward masks of the model this cache belongs to.
ReadonlystartPositions held in bf16 before the conversion; 0 converts after the first append.
ReadonlyturboHow the converted storage attends, named by the graph the cache serves
(denseTurboQuantKernels for Qwen 3.5, deferredTurboQuantKernels for
Gemma 4 and MiniCPM5).
ReadonlyvThe donor reads of the storage held now: a drafter borrowing this layer's keys and values reads them as the storage hands them out.
Earliest prefix whose retained representation can resume generation. Irreversible transitions may preserve bytes while invalidating older precision boundaries. Reuse policy must not trim below this offset.
Earliest prefix whose retained representation can resume generation. Irreversible transitions may preserve bytes while invalidating older precision boundaries. Reuse policy must not trim below this offset.
state() returns fresh views once the storage is encoded.
Bidirectional window: the same as appendWindow, except that two
positions both flagged in bidirectional see each other in either
direction, regardless of order or the sliding window. bidirectional is a
borrowed bool [L] that applies to every row. Defined only for the first
window of a sequence (offset 0).
Committed span: L decided positions per row, causal within the span.
Each query attends its own causal prefix with the arithmetic of a
one-position read, so the span keeps the single-position reduction order
the graph qualifies per KV format (MlxKnownSpans, read by appendKnown). This replaces the
independentPositions flag. The only caller today runs one row with up to
four positions.
Decode: one new position per row (L = 1). Each row's query sees that
row's valid stored positions up to and including the one just appended,
limited to the cache's sliding window.
The encoder window, a first window: bf16 storage's read, then the transition its append owes.
Window: L new positions per row, causal within the window, for prefill
chunks, prefill tails and speculative verify windows. Query i of a row
sees the row's earlier valid positions and window positions 0..i, limited
to the cache's sliding window. Left padding and prepared prefill padding are
never visible. Outputs at padded query positions are unspecified and the
caller ignores them. To append context without attending, make this call
and dispose the read unused.
Physical cache storage consumed by one additional token for one row. Recurrent caches return 0 because their state does not grow with the sequence. Optional for stateless/training adapters that are never admitted or persisted.
Apply the closure's outputs (takes ownership); returns arrays to evaluate with the step.
The closure inputs of slot, in order.
Non-mutating: the fetch prepareDecodeStep will choose.
Releases the storage and returns to the constructed state: empty bf16.
Can trim(n) drop the last n tokens? (Ring caches lose trimability
once wrapped.)
Maximum committed positions before this state changes precision.
Host-side bookkeeping for a 1-token step (growth, rotation), without the write.
The block read of the storage held now (it appends nothing, so nothing converts).
Stable storage identity for compatibility guards and persistence. REQUIRED: every capability predicate (isPlainKvCache, isRotating*, packages/inference/src/state/persistence.ts codecs) is a string compare on this value, so a missing override used to fail OPEN into wrong routing (PRs #42/#43: BatchedRotatingCache shipped without one and batched joins dropped running rows). Wrappers report their own kind plus the inner kind.
Drop the last n tokens (future writes overwrite them). bypass (default
false → unchanged behavior) skips the isTrimmable guard for the
speculative-decode rollback case (trim the last n ≤ γ tips right after a
1 concat write); rotating caches then physically shrink the buffer tail.
TurboQuant-encoded keys and values for one full-attention layer, bf16 first: the lego
--kv-quant turbocomposes (sliding-window layers keep a plainRotatingKVCache).Numbers: today's served path. Storage is a
KVCacheuntil an append leaves it holding at leaststartpositions (the first append whenstartis 0). That append is read bf16 and then, before it returns, the storage converts (TurboQuantKVCache.fromKVCacheover the live window, the reuse floor at a nonzero start, materialized at once). Every later append encodes its positions and is read through the codec. Stored values below the offset equal the convert-after path's bit for bit. The transition has the obligations described on Bf16FirstQuantizedKVCache.Reads: before the transition,
KVCache's reads; after it, the encoded storage's reads, which attend withturboQuantKernels(no mask for decode, the causal mask for a window): decoded values and the stock fused SDPA for Qwen 3.5's graph, or the deferred-V fetch, the stock fused SDPA and the inverse rotation of the output for Gemma 4's and MiniCPM5's.fusedDecodeis resolved by whoever builds the cache; nothing here reads the runtime configuration.