The affine kernel set converted rings and every read attend with,
named by the composition (composedAffineKernels).
Optionalrow: CacheOptionalcompose: (storage: Cache) => Cache
How the lego these rings came from composes the rings this layout
hands back to serial use (ComposedStorage.over).
ReadonlybitsOptional ReadonlycomposeHow the lego these rings came from composes the rings this layout
hands back to serial use (ComposedStorage.over).
ReadonlygroupReadonlykernelThe affine kernel set converted rings and every read attend with,
named by the composition (composedAffineKernels).
ReadonlymaxReadonlyspeculativeReadonlystartReadonlystatestate() returned temporary views that the caller must release.
ReadonlytransitionEarliest prefix whose retained representation can resume generation. Irreversible transitions may preserve bytes while invalidating older precision boundaries. Reuse policy must not trim below this offset.
ProtectedpackedProtectedpositionsA preparation method commits each request's own precision boundary. Splitting a forward for a sibling must not move that boundary.
Compiled-decode trace adapters expose the RoPE offset as an int32 array input here; real caches leave it unset (static int path).
ProtectedrowsProtectedadvanceThe committed read (Qwen 3.5's appendKnown under knownSpans): the same
mask and attention view, attended with independent positions, which the
packed layout's view honours and the per-row views ignore.
Decode: one new position per row (L = 1). Each row's query sees that
row's valid stored positions up to and including the one just appended,
limited to the cache's sliding window.
Window: L new positions per row, causal within the window, for prefill
chunks, prefill tails and speculative verify windows. Query i of a row
sees the row's earlier valid positions and window positions 0..i, limited
to the cache's sliding window. Left padding and prepared prefill padding are
never visible. Outputs at padded query positions are unspecified and the
caller ignores them. To append context without attending, make this call
and dispose the read unused.
Physical cache storage consumed by one additional token for one row. Recurrent caches return 0 because their state does not grow with the sequence. Optional for stateless/training adapters that are never admitted or persisted.
Apply the closure's outputs (takes ownership); returns arrays to evaluate with the step.
The closure inputs of slot, in order.
The fetch of the coming compiled step: the lego's while it holds the row,
else the rows' (decodePhaseOfRows), without moving the row.
ProtecteddecodeA ring's lone row fetches by its ring: concat until the next write lands at
the window, the ring after (RotatingKVCache.decodePhase).
A row handed back to serial use: its storage, composed again by the lego
the row came from (KvRowTransition.compose).
A row's storage, owned by the caller: what another row layout merges.
Can trim(n) drop the last n tokens? (Ring caches lose trimability
once wrapped.)
Maximum committed positions before this state changes precision.
Host-side bookkeeping for a 1-token step (growth, rotation), without the write.
Stable storage identity for compatibility guards and persistence. REQUIRED: every capability predicate (isPlainKvCache, isRotating*, packages/inference/src/state/persistence.ts codecs) is a string compare on this value, so a missing override used to fail OPEN into wrong routing (PRs #42/#43: BatchedRotatingCache shipped without one and batched joins dropped running rows). Wrappers report their own kind plus the inner kind.
Speculative verify-round support for NON-trimmable recurrent caches
(SSMCache — gated-DeltaNet conv + recurrent state). A cache exposing
these is spec-eligible without isTrimmable(): the serve loop arms a
round before the verify forward (the layer snapshots pre-round state —
free, MLX arrays are immutable — and records its position-local kernel
inputs), then resolves it: commit on full accept, or rollback(keep)
which restores the snapshot and bit-exactly REPLAYS the first keep
window tokens through the recurrence (identical arithmetic prefix ⇒
identical state). Trimmable caches leave these unset and keep trim().
ProtectedsyncDrop the last n tokens (future writes overwrite them). bypass (default
false → unchanged behavior) skips the isTrimmable guard for the
speculative-decode rollback case (trim the last n ≤ γ tips right after a
1 concat write); rotating caches then physically shrink the buffer tail.
The dense fetch (the softcapped read's): after the transition the append runs, each row's own ring answers as its storage fetches it dense (a converted ring dequantized; one row keeps its physical columns and phase; several share one aligned geometry) and rows join along the batch, or the packed layout's dense fetch once every row is converted; the caller owns them.
The rows of the rotating affine bf16-first lego (
Bf16FirstRotatingQuantizedKVCache) for continuous batching. Precision changes preserve each row's physical columns. The model owns queries and scale; the shared lifecycle owns membership and the transition (when a row converts is described onTransitioningKvRows). AstartofInfinitynever converts: the prefill cohort's layout for plain rings. The reads are the graph's call today: this layout's mask before the append, under the ring's window, then its attention view. Converted rings and the packed layout attend withkernelSet, taken when the layout is built; the rows a cohort extracts carry it into the layouts they join.