Tokens this cache must be able to hold (prompt + maxTokens).
Optionalquantization: PagedQuantization
Affine pages; absent with turbo absent: bf16 pages.
The masks the bidirectional and encoder reads lease.
Optionalturbo: PagedTurboQuant
TurboQuant pages.
ReadonlyblockLogical block order → physical pool slot.
ReadonlycapacityTokens this cache must be able to hold (prompt + maxTokens).
ReadonlydirectThe direct Metal read, where the page format takes it (see the class).
ReadonlymasksThe masks the bidirectional and encoder reads lease.
Optional ReadonlyquantizationAffine pages; absent with turbo absent: bf16 pages.
Optional ReadonlyropeSet only by compiled-decode trace adapters (see Cache); a real cache leaves it unset.
Optional ReadonlyturboTurboQuant pages.
Static ReadonlyDEFAULT_Matches KVCache.STEP: v1's growth granularity is a permutation of today's 256-token step into fixed reusable slots, not a new tuning axis (--paged-kv-block-size overrides for experiments).
KVCache's vision prefill read: the causal mask OR'd with the image
block overlay, over the gathered first window.
The window read: KVCache's committed read, over the same positions.
Decode: one new position per row (L = 1). Each row's query sees that
row's valid stored positions up to and including the one just appended,
limited to the cache's sliding window.
KVCache's encoder read: the encoder window's mask over the gathered
first window.
Window: L new positions per row, causal within the window, for prefill
chunks, prefill tails and speculative verify windows. Query i of a row
sees the row's earlier valid positions and window positions 0..i, limited
to the cache's sliding window. Left padding and prepared prefill padding are
never visible. Outputs at padded query positions are unspecified and the
caller ignores them. To append context without attending, make this call
and dispose the read unused.
The gathered positions, as KVCache hands its donor rows out.
The write half (taking ownership of outputs): bf16 and affine pages
append the step's row as an ordinary append does; TurboQuant pages store
the step's codes, packed as the codec packs them. Returns the pages to
evaluate with the step.
The stored positions gathered (dequantized on affine pages, decoded in the kernel set's domain on TurboQuant pages), owned by the step.
Non-mutating: the fetch prepareDecodeStep will choose.
TurboQuant pages take TurboQuant storage's slot (the codes leave the graph); bf16 and affine pages the paged one (the row leaves it).
Can trim(n) drop the last n tokens? (Ring caches lose trimability
once wrapped.)
Same mask policy as KVCache.makeMask (it reads only offset).
Compiled decode: nothing to prepare on the host; the pages grow when the step's row is appended.
KVCache's block read: the gathered positions in order, then the block's
keys and values, with no mask. Nothing is appended.
Distinct storage kind; the row adapter preserves its block pools.
Owned planes materialized at prefill and immutable publication boundaries.
Rewind the last n tokens and return now-unoccupied tail blocks to the free list. Stale bytes past offset are never read (fetch slices to offset; writes cover exact ranges) — the KVCache padding invariant, block-shaped.
Cache implementation over a BlockPool. Same values as a plain KVCache at every step (parity-gated); only the physical arrangement differs. The pool allocates lazily on the first write (KVCache's lazy-alloc shape): head count / head dims / dtype come from the first k/v pair, so the wiring (maybePageKv) needs no per-model shape plumbing.
Reads: each owns an immutable view of the blocks as of its append (
pagedAttentionView). Decode attends its one query, and a window its queries under the causal mask, with the readdirect(the composition'spagedKv.direct) fixes at construction: the direct Metal read from the blocks for at most eight queries, or the gathered blocks through the stock fused SDPA (decode with no mask, a window with the causal mask). A window of more than eight queries gathers under either.The other reads are
KVCache's over the gathered blocks, so bf16 pages read bit for bit as the bf16 lego they replace: a committed span is the window read (asKVCachequalifies it); the bidirectional and encoder windows append, gather and attend underKVCache's masks; a block read attends the gathered positions and the block's own keys with no mask; the donor rows are the gathered positions. Affine pages gather dequantized.TurboQuant pages read as
TurboQuantKVCachereads its storage: the window decoded per position by the codec, in the domain the layer's kernel set reads (PagedTurboQuant.kernels), attended by that set (no mask for decode, the causal mask for a window); the other reads take the window decoded to the values' own domain. The direct read takes bf16 pages and affine widths that divide 32 bits; other pages read gathered whatever the composition says.Compiled decode, gathered pages: a concat slot. The stored positions are gathered (and dequantized) outside the recorded graph, as the decode read gathers them; in it the new position is stored as the pages store it (quantized and dequantized again on affine pages) and joined to them, and the fused SDPA reads the joined window with no mask, the gathered decode read. The row leaves the graph as it was appended; the write after the step appends it to the pages. Direct pages have no compiled form (
pagedDecodeCompiles): their read is a custom kernel whose partitions follow the context length, which a shapeless recorded graph cannot hold, so their composition decodes eagerly and this cache refuses a compiled slot.