The per-forward masks of the model this layout serves.
ReadonlymasksThe per-forward masks of the model this layout serves.
Readonlymaxmlx-lm uses cache.offset (the per-row array) for the scalar interface
too; we expose the scalar total as offset and the per-row positions via
ropeOffsetArr (the model's per-row RoPE path).
Compiled-decode trace adapters expose the RoPE offset as an int32 array input here; real caches leave it unset (static int path).
Decode: one new position per row (L = 1). Each row's query sees that
row's valid stored positions up to and including the one just appended,
limited to the cache's sliding window.
Window: L new positions per row, causal within the window, for prefill
chunks, prefill tails and speculative verify windows. Query i of a row
sees the row's earlier valid positions and window positions 0..i, limited
to the cache's sliding window. Left padding and prepared prefill padding are
never visible. Outputs at padded query positions are unspecified and the
caller ignores them. To append context without attending, make this call
and dispose the read unused.
mlx-lm BatchRotatingKVCache.extract (models/cache.py:1417): row i
as a fresh SERIAL RotatingKVCache — de-rolled to temporal order (the
oracle's roll(-_idx) when rotated), left padding stripped
(max(0, left_padding[idx]) — post-wrap pads go negative), OWNED
contiguous copies. offset = the row's absolute position (may exceed
the buffer: wrapped rings have evicted tokens); ring idx = the new
buffer length — temporal order ⇔ write head at the end, exactly the
oracle's cache._idx = cache.keys.shape[2]. Bit-exact vs a solo run:
merge/decode/filter keep each row's ring bytes identical to the serial
cache's (02d723a:tests/unit/batched-rotating.test.ts) and this is a pure
slice+copy.
Optionallimit: numberKeep only keep rows along the batch axis (eviction).
Can trim(n) drop the last n tokens? (Ring caches lose trimability
once wrapped.)
Join rows into this empty ring: sources are running rings, whose rows are cut out in temporal order, or serial RotatingKVCaches (an adopted lone row or a joiner), in row order. Sources are borrowed.
OptionalrightPaddedBatch: booleanBytes one row holds for tokens more positions, capped at the window.
Free the per-step RoPE array without disposing KV (wrapper-rebuild path).
Same signature as the serial RotatingKVCache — the scheduler's merge guards recognize a batched ring as "rotating-plain + RowBatchCache" (packages/inference/src/execution/batch-group.ts #mergeJoiner prevRot branch); without this the next join sees "unknown", skips the prevRot rows, and replaces the batch's sliding-layer KV with a one-row cache (B≥3 collapse). Guard order everywhere is isRowBatchCache FIRST, so the shared string never misroutes a batched cache into a serial-only path.
Drop the last n tokens (future writes overwrite them). bypass (default
false → unchanged behavior) skips the isTrimmable guard for the
speculative-decode rollback case (trim the last n ≤ γ tips right after a
1 concat write); rotating caches then physically shrink the buffer tail.
StaticadoptAdopt owned tensors without changing their physical ring columns.
StaticmergeAssemble a batch from per-row temporal KV slices (port of merge). Each row
is its solo cache's temporalView ([1,H,Li,D], Li ≤ maxSize, temporal
order) and offsets[i] its absolute position. Left-pads to the longest
row; the result is in temporal order (rotated=false, idx=offset=width).
Port of mlx-lm BatchRotatingKVCache; padded singleton chunks use its concat operation with the matching block mask until finalization.
Reads: the sliding window is
maxSize. Decode, which runs with no prefill padding pending (finalize first), writes one position in place at the ring head; a window writes by concatenation, which also serves padded prefill chunks of any length. Each attends with the stock fused SDPA under mlx-lm's batched rotating mask (buildBatchedRotatingMask) for its write, built once per forward throughmasks. The deprecatedupdateAndFetchpicks the write from the length instead, so a one-row window without pending padding wrote in place there and concatenates here: the same bits until the ring wraps, a different key order after.