The sliding window: the ring's capacity and the window every read applies.
Positions held in bf16 before the conversion; 0 converts after the first append.
The affine kernel set the converted storage attends with, named by the
composition (composedAffineKernels).
The per-forward masks of the model this cache belongs to.
Optionalstorage: CacheReadonlybitsReadonlycomposedThe storage the row layouts, clones and saved files take.
ReadonlygroupReadonlykernelThe affine kernel set the converted storage attends with, named by the
composition (composedAffineKernels).
ReadonlymasksThe per-forward masks of the model this cache belongs to.
ReadonlymaxThe sliding window: the ring's capacity and the window every read applies.
ReadonlystartPositions held in bf16 before the conversion; 0 converts after the first append.
The donor reads of the storage held now: a drafter borrowing this layer's keys and values reads them as the storage hands them out.
Earliest prefix whose retained representation can resume generation. Irreversible transitions may preserve bytes while invalidating older precision boundaries. Reuse policy must not trim below this offset.
Earliest prefix whose retained representation can resume generation. Irreversible transitions may preserve bytes while invalidating older precision boundaries. Reuse policy must not trim below this offset.
Bidirectional window: the same as appendWindow, except that two
positions both flagged in bidirectional see each other in either
direction, regardless of order or the sliding window. bidirectional is a
borrowed bool [L] that applies to every row. Defined only for the first
window of a sequence (offset 0).
A committed span, one decode per position through this lego
(decodeByPosition), so the transition falls where L decodes put it.
Decode: one new position per row (L = 1). Each row's query sees that
row's valid stored positions up to and including the one just appended,
limited to the cache's sliding window.
The encoder window, a first window: bf16 storage's read, then the transition its append owes.
Window: L new positions per row, causal within the window, for prefill
chunks, prefill tails and speculative verify windows. Query i of a row
sees the row's earlier valid positions and window positions 0..i, limited
to the cache's sliding window. Left padding and prepared prefill padding are
never visible. Outputs at padded query positions are unspecified and the
caller ignores them. To append context without attending, make this call
and dispose the read unused.
Physical cache storage consumed by one additional token for one row. Recurrent caches return 0 because their state does not grow with the sequence. Optional for stateless/training adapters that are never admitted or persisted.
Apply the closure's outputs (takes ownership); returns arrays to evaluate with the step.
The closure inputs of slot, in order.
Non-mutating: the fetch prepareDecodeStep will choose.
Releases the storage and returns to the constructed state: empty bf16.
Can trim(n) drop the last n tokens? (Ring caches lose trimability
once wrapped.)
Maximum committed positions before this state changes precision.
Host-side bookkeeping for a 1-token step (growth, rotation), without the write.
The block read of the storage held now (it appends nothing, so nothing converts).
Stable storage identity for compatibility guards and persistence. REQUIRED: every capability predicate (isPlainKvCache, isRotating*, packages/inference/src/state/persistence.ts codecs) is a string compare on this value, so a missing override used to fail OPEN into wrong routing (PRs #42/#43: BatchedRotatingCache shipped without one and batched joins dropped running rows). Wrappers report their own kind plus the inner kind.
Drop the last n tokens (future writes overwrite them). bypass (default
false → unchanged behavior) skips the isTrimmable guard for the
speculative-decode rollback case (trim the last n ≤ γ tips right after a
1 concat write); rotating caches then physically shrink the buffer tail.
Optionalbypass: boolean
Affine-quantized keys and values for one sliding-window layer, bf16 first: the rotating counterpart of Bf16FirstQuantizedKVCache, with the same numbers, transition and obligations. Storage is a
RotatingKVCacheofmaxSizeuntil an append leaves it holding at leaststartpositions, then converts (RotatingKVCache.toQuantized: the ring as laid out, its write index carried) into aRotatingQuantizedKVCache, whose own reads serve every later append. Every written ring position equals the convert-after path's bit for bit.