plain: the composition's affine kernel set (composedAffineKernels); decodeBlock: the decode kernel's
tokens per threadgroup its device's table names (the M4 Pro's by default).
Optionalstorage: CacheReadonlybitsReadonlycomposedThe storage the row layouts, clones and saved files take (see the class).
ReadonlydecodeReadonlygroupReadonlykernelThe affine kernel set the converted storage attends with, named by the
composition (composedAffineKernels, or the KV4 set).
ReadonlymasksThe per-forward masks of the model this cache belongs to.
ReadonlyplainReadonlystartPositions held in bf16 before the conversion; 0 converts after the first append.
The donor reads of the storage held now: a drafter borrowing this layer's keys and values reads them as the storage hands them out.
Earliest prefix whose retained representation can resume generation. Irreversible transitions may preserve bytes while invalidating older precision boundaries. Reuse policy must not trim below this offset.
Earliest prefix whose retained representation can resume generation. Irreversible transitions may preserve bytes while invalidating older precision boundaries. Reuse policy must not trim below this offset.
Bidirectional window: the same as appendWindow, except that two
positions both flagged in bidirectional see each other in either
direction, regardless of order or the sliding window. bidirectional is a
borrowed bool [L] that applies to every row. Defined only for the first
window of a sequence (offset 0).
Committed span: L decided positions per row, causal within the span.
Each query attends its own causal prefix with the arithmetic of a
one-position read, so the span keeps the single-position reduction order
the graph qualifies per KV format (MlxKnownSpans, read by appendKnown). This replaces the
independentPositions flag. The only caller today runs one row with up to
four positions.
Decode: one new position per row (L = 1). Each row's query sees that
row's valid stored positions up to and including the one just appended,
limited to the cache's sliding window.
The encoder window, a first window: bf16 storage's read, then the transition its append owes.
Window: L new positions per row, causal within the window, for prefill
chunks, prefill tails and speculative verify windows. Query i of a row
sees the row's earlier valid positions and window positions 0..i, limited
to the cache's sliding window. Left padding and prepared prefill padding are
never visible. Outputs at padded query positions are unspecified and the
caller ignores them. To append context without attending, make this call
and dispose the read unused.
Physical cache storage consumed by one additional token for one row. Recurrent caches return 0 because their state does not grow with the sequence. Optional for stateless/training adapters that are never admitted or persisted.
Apply the closure's outputs (takes ownership); returns arrays to evaluate with the step.
The closure inputs of slot, in order.
Non-mutating: the fetch prepareDecodeStep will choose.
Releases the storage and returns to the constructed state: empty bf16.
Can trim(n) drop the last n tokens? (Ring caches lose trimability
once wrapped.)
Positions an append may add before the transition: up to the start.
Host-side bookkeeping for a 1-token step (growth, rotation), without the write.
The block read of the storage held now (it appends nothing, so nothing converts).
Stable storage identity for compatibility guards and persistence. REQUIRED: every capability predicate (isPlainKvCache, isRotating*, packages/inference/src/state/persistence.ts codecs) is a string compare on this value, so a missing override used to fail OPEN into wrong routing (PRs #42/#43: BatchedRotatingCache shipped without one and batched joins dropped running rows). Wrappers report their own kind plus the inner kind.
Drop the last n tokens (future writes overwrite them). bypass (default
false → unchanged behavior) skips the isTrimmable guard for the
speculative-decode rollback case (trim the last n ≤ γ tips right after a
1 concat write); rotating caches then physically shrink the buffer tail.
The 4-bit group-64 full-attention lego of the Qwen3.8-27B Trellis M4 Pro and M1 Max compositions, bf16 first: Bf16FirstQuantizedKVCache whose quantized storage reads with
Kv4Head256Cache's kernels, the decode kernel at the composition'sdecodeBlock. The first append is bf16, as the old graph's conversion after the first forward kept it; every later decode reads with the fused one-row KV4 kernel and every window with the folded-GQA kernel, the cores the old graph injected for one sequence. Its storage carries the KV4 set into the row layouts it joins, which read with the KV4 kernels while they hold one unpadded row and withplain's affine kernels when they hold more (AffineKernels.multiRow), as the old graph read several sequences through the generic path.