ReadonlybitsReadonlygroupReadonlykernelsThe kernel of each read.
ReadonlymasksThe per-forward masks of the model this cache belongs to. Row layouts
made from this cache (state/layout) share them.
ReadonlymaxOptionalminimumEarliest prefix whose retained representation can resume generation. Irreversible transitions may preserve bytes while invalidating older precision boundaries. Reuse policy must not trim below this offset.
Ring write index (oracle _idx); public so toQuantized replay
and persistence can carry it.
Per-row RoPE positions. STABLE ACROSS A STEP — refreshed only at releaseRopeArr() (the scheduler's post-dispatch hook), never inside the update: a graph that captures it before the update ropes Q after (an in-update dispose would be a use-after-dispose), and one that re-reads it for Q after the update would see K and Q at different positions after an in-update refresh. Widens the base's readonly to mutable.
Static ReadonlySTEPThe kernel set affine storage restored at this cache's layer reads with
(state/persistence): this cache's own kernels.
A committed span, one decode per position (decodeByPosition): the
in-place ring write and the decode kernel each position would get.
Decode: one new position per row (L = 1). Each row's query sees that
row's valid stored positions up to and including the one just appended,
limited to the cache's sliding window.
The mask is makeMask's with the window at maxSize, before the append:
plain causal while the window has room, the window-bound array after.
The window write (concatenation) attended with kernel under mask,
for the layout that owns this ring and knows its mask is plain causal
(SpeculativeRotatingAffineLayout). Takes ownership of mask.
Physical cache storage consumed by one additional token for one row. Recurrent caches return 0 because their state does not grow with the sequence. Optional for stateless/training adapters that are never admitted or persisted.
The live window as donor attention (see temporalView). Caller owns the returned views.
Apply the closure's outputs (takes ownership); returns arrays to evaluate with the step.
The closure inputs of slot, in order.
Non-mutating twin of prepareDecodeStep's fetch choice (see RotatingKVCache).
Quantized twin of BatchedRotatingCache.extractRow (mlx-lm
BatchRotatingKVCache.extract, models/cache.py:1417, over triples):
row i as a fresh SERIAL RotatingQuantizedKVCache — de-rolled to
temporal order, left padding stripped, OWNED contiguous copies;
offset = the row's absolute position, ringIdx = the new buffer length
(the oracle's cache._idx = cache.keys.shape[2]). Token-axis slicing
is byte-safe (packing along HEAD_DIM — file header); per-row byte
identity vs the serial oracle is the class invariant
(packages/inference/tests/state/batched-rotating-quant.test.ts), extraction
is a pure slice+copy.
Keep only keep rows along the batch axis (eviction), in place.
Ring rule (inherited semantics): trimmable only before wrap.
Same mask formula as RotatingKVCache (inherited in the oracle).
Join rows into this empty ring: sources are running rings, whose rows are cut out in temporal order, or serial RotatingQuantizedKVCaches (an adopted lone row or a joiner), in row order. Sources are borrowed.
Compiled decode: host-side bookkeeping of #updateInPlace (L=1) — growth, oversize trim, rotation — without quantize/write.
OptionalrightPaddedBatch: booleanBytes one row holds for tokens more positions, capped at the window.
A block read (BlockAttentionCache.readBlock) as RotatingKVCache
makes it: the live window in temporal order, cut to the newest
maxSize - 1 positions once it is full, then the block's keys and values
quantized as this ring stores them and not appended, under the same mask
(rotatingBlockMask), by the masked-window kernel.
Scheduler hook, called on every fwd cache AFTER the step's graph is built — the ONE safe point to swap the rope array for the advanced positions (the step's graph holds its own reference; no JS reads the old handle after this).
Adopt persisted state (takes ownership of the triples' arrays) — ring order as-laid-out, ringIdx carried with it (packages/inference/src/state/persistence.ts).
Stable storage identity for compatibility guards and persistence. REQUIRED: every capability predicate (isPlainKvCache, isRotating*, packages/inference/src/state/persistence.ts codecs) is a string compare on this value, so a missing override used to fail OPEN into wrong routing (PRs #42/#43: BatchedRotatingCache shipped without one and batched joins dropped running rows). Wrappers report their own kind plus the inner kind.
Ring contents in temporal order, cut to the valid length (extract) — overrides the base to use the BATCHED ring state.
Oracle: to_quantized on an already-quantized rotating cache is idempotent.
Drop the last n tokens (future writes overwrite them). bypass (default
false → unchanged behavior) skips the isTrimmable guard for the
speculative-decode rollback case (trim the last n ≤ γ tips right after a
1 concat write); rotating caches then physically shrink the buffer tail.
Quantize incoming k/v and write into the ring; returns active quantized triples (S=1 in place, S>1 via temporal-order + concat).
StaticadoptAdopt owned tensors without changing their physical ring columns.
Staticemptykernels: the composition's kernel set for these bits and group size.
masks: the per-forward masks of the model this layout serves.
StaticmergeAssemble a batch from per-row temporal quantized slices (the bf16
twin's merge over triples). Each row is a solo/adopted/batched cache's
temporalView ([1,H,Li,*], Li ≤ maxSize, temporal order) and
offsets[i] its absolute position. Result is in temporal order
(rotated=false, ringIdx=offset=width).
Batched ring over quantized triples — mlx-lm BatchRotatingKVCache mechanics (see batched-rotating.ts) with RotatingQuantizedKVCache storage. Scalar ring state (
ringIdx/offsetreuse the base fields); per-rowoffsetArr(absolute positions → RoPE) andleftPad.Reads: the sliding window is
maxSize. Decode, which runs with no prefill padding pending (finalize first), writes one position in place at the ring head; a window writes by concatenation, which also serves padded prefill chunks. Each attends under mlx-lm's batched rotating mask (buildBatchedRotatingMask) for its write, built once per forward throughmasks, with the decode and masked-window kernels ofkernels: that mask is always an array, whichquantizedSdpanever tiles, so the unfused kernel both composed sets hold there is what every configuration computes today. A row it extracts attends alone with the samekernels. The deprecatedupdateAndFetchQuantizedpicks the write from the length instead, so a one-row window without pending padding wrote in place there and concatenates here: the same bits until the ring wraps, a different key order after.