The kernel of each read.
The per-forward masks of the model this cache belongs to. Row layouts
made from this cache (state/layout) share them.
ReadonlybitsReadonlygroupReadonlykernelsThe kernel of each read.
ReadonlymasksThe per-forward masks of the model this cache belongs to. Row layouts
made from this cache (state/layout) share them.
ReadonlymaxOptionalminimumEarliest prefix whose retained representation can resume generation. Irreversible transitions may preserve bytes while invalidating older precision boundaries. Reuse policy must not trim below this offset.
Ring write index (oracle _idx); public so toQuantized replay
and persistence can carry it.
Optional ReadonlyropeSet only by compiled-decode trace adapters (see Cache).
Static ReadonlySTEPThe kernel set affine storage restored at this cache's layer reads with
(state/persistence): this cache's own kernels.
A committed span, one decode per position (decodeByPosition): the
in-place ring write and the decode kernel each position would get.
Decode: one new position per row (L = 1). Each row's query sees that
row's valid stored positions up to and including the one just appended,
limited to the cache's sliding window.
The mask is makeMask's with the window at maxSize, before the append:
plain causal while the window has room, the window-bound array after.
Physical cache storage consumed by one additional token for one row. Recurrent caches return 0 because their state does not grow with the sequence. Optional for stateless/training adapters that are never admitted or persisted.
The live window as donor attention (see temporalView). Caller owns the returned views.
Apply the closure's outputs (takes ownership); returns arrays to evaluate with the step.
The closure inputs of slot, in order.
Non-mutating twin of prepareDecodeStep's fetch choice (see RotatingKVCache).
Ring rule (inherited semantics): trimmable only before wrap.
Same mask formula as RotatingKVCache (inherited in the oracle).
Compiled decode: host-side bookkeeping of #updateInPlace (L=1) — growth, oversize trim, rotation — without quantize/write.
A block read (BlockAttentionCache.readBlock) as RotatingKVCache
makes it: the live window in temporal order, cut to the newest
maxSize - 1 positions once it is full, then the block's keys and values
quantized as this ring stores them and not appended, under the same mask
(rotatingBlockMask), by the masked-window kernel.
Adopt persisted state (takes ownership of the triples' arrays) — ring order as-laid-out, ringIdx carried with it (packages/inference/src/state/persistence.ts).
Stable storage identity for compatibility guards and persistence. REQUIRED: every capability predicate (isPlainKvCache, isRotating*, packages/inference/src/state/persistence.ts codecs) is a string compare on this value, so a missing override used to fail OPEN into wrong routing (PRs #42/#43: BatchedRotatingCache shipped without one and batched joins dropped running rows). Wrappers report their own kind plus the inner kind.
The live window in chronological order: the newest min(offset, maxSize) positions, the quantized twin of RotatingKVCache.temporalView (the tail of an oversized multi-token block, never its head).
Oracle: to_quantized on an already-quantized rotating cache is idempotent.
Drop the last n tokens (future writes overwrite them). bypass (default
false → unchanged behavior) skips the isTrimmable guard for the
speculative-decode rollback case (trim the last n ≤ γ tips right after a
1 concat write); rotating caches then physically shrink the buffer tail.
Quantize incoming k/v and write into the ring; returns active quantized triples (S=1 in place, S>1 via temporal-order + concat).
Quantized rotating (sliding-window) KV cache — port of optiq runtime/kv/rotating.py RotatingQuantizedKVCache with keep=0 (gemma4's configuration): RotatingKVCache's ring mechanics over (packed, scales, biases) triples, storage convention identical to QuantizedKVCache. Returns ACTIVE QUANTIZED SLICES — the oracle's module docstring claims dequantize-on-read, but its code does not (port follows the code). optiq's producer-registry
Built empty, it quantizes from token zero, unlike the served path's
Bf16FirstRotatingQuantizedKVCache, so its first chunk's numbers differ.Reads: the sliding window is
maxSize, the window every graph passes for the layers it gives this cache. Decode writes in place at the ring head and attends the whole ring with no mask (eviction enforces the window). A window writes by concatenation, in temporal order, under the bottom-right causal matrix with the window kernel, orcreateCausalMaskwith the sliding window and the masked-window kernel once the window binds. The kernels are fixed at construction. Today the graphs'updateAndFetchQuantizedpicks the in-place write for one position andquantizedSdpapicks the unfused port for one query, so a one-row window concatenates here (the same keys, in another order once the ring wraps) and takes the window kernel.