How the named reads attend, named by the graph the cache serves; copies, row layouts and extracted rows keep it.
The per-forward masks of the model this cache belongs to. Row layouts
made from this cache (state/layout) share them.
ReadonlyfusedReadonlykReadonlymasksThe per-forward masks of the model this cache belongs to. Row layouts
made from this cache (state/layout) share them.
Earliest prefix whose retained representation can resume generation. Irreversible transitions may preserve bytes while invalidating older precision boundaries. Reuse policy must not trim below this offset.
Optional ReadonlyropeSet only by compiled-decode trace adapters (see Cache); a real cache leaves it unset.
Readonlystatestate() returned temporary views that the caller must release.
ReadonlyturboHow the named reads attend, named by the graph the cache serves; copies, row layouts and extracted rows keep it.
ReadonlyvStatic ReadonlySTEPThe window read: a committed span on this storage (Qwen 3.5's
appendKnown) attends with the same call as a window.
Decode: one new position per row (L = 1). Each row's query sees that
row's valid stored positions up to and including the one just appended,
limited to the cache's sliding window.
Window: L new positions per row, causal within the window, for prefill
chunks, prefill tails and speculative verify windows. Query i of a row
sees the row's earlier valid positions and window positions 0..i, limited
to the cache's sliding window. Left padding and prepared prefill padding are
never visible. Outputs at padded query positions are unspecified and the
caller ignores them. To append context without attending, make this call
and dispose the read unused.
Physical cache storage consumed by one additional token for one row. Recurrent caches return 0 because their state does not grow with the sequence. Optional for stateless/training adapters that are never admitted or persisted.
The write half: pack the step's codes as the codec packs them and store
them at the offset. Takes ownership of outputs; returns the storage to
evaluate with the step.
The stored prefix, decoded as the kernel set's read decodes it (owned by the step).
Non-mutating: the fetch prepareDecodeStep will choose.
Can trim(n) drop the last n tokens? (Ring caches lose trimability
once wrapped.)
Same rule as KVCache/QuantizedKVCache.makeMask (mlx-lm cache.py:114-125): windowless multi-token chunks are the string "causal" at any offset — TurboQuant v1 is full-attention-only, never windowed.
Compiled decode: the growth half of a one-position append, without the write.
A block read (BlockAttentionCache.readBlock): the stored positions
decoded, then the block's keys and values, not appended, through the stock
fused SDPA with no mask (the dense read, whichever kernel set the cache
composes, since the block's values are not in the rotated domain).
Adopt persisted/converted state (takes ownership of the arrays).
Stable storage identity for compatibility guards and persistence. REQUIRED: every capability predicate (isPlainKvCache, isRotating*, packages/inference/src/state/persistence.ts codecs) is a string compare on this value, so a missing override used to fail OPEN into wrong routing (PRs #42/#43: BatchedRotatingCache shipped without one and batched joins dropped running rows). Wrappers report their own kind plus the inner kind.
Trimmed-to-offset arrays in documented order: kIdx, kScales, kZeros, vPacked, vScales (packages/inference/src/state/persistence.ts tensor-slot ordering for kind "turboquant"). Empty cache → [].
Drop the last n tokens (byte-safe: groups run along head_dim, so a token-axis slice never splits a group). Same front/back convention as KVCache.trim: offset shrinks, storage is overwritten on next write.
Deferred-inverse-FWHT read path: same append, but the returned V window stays in the ROTATED domain. The caller MUST un-rotate its attention output with tq.unrotateValues (see SharedKv.vRotated) — same decode by linearity, O(q·d log d) per step instead of O(T·d log d). Opt-in per attention site; sites that don't know about rotation keep calling updateAndFetch and stay correct.
StaticfromConvert an existing bf16 KVCache/RotatingKVCache's live window in one
shot: quantize the whole [.., :offset, :] region, preserve offset,
dispose the source's arrays. Mirrors KVCache.toQuantized/
RotatingKVCache.toQuantized's contract (source cache is consumed).
kernels is the converted cache's read: for a graph's KVCache, the read
the graph composed into it (KVCache.turboQuantKernels). fusedDecode is
the converted cache's decode policy; a caller that has resolved it passes
it, the default reads the runtime configuration.
TurboQuant KV cache (
02d723a:docs/design/turboquant.md): encoded at rest, decoded by its named reads with the kernel set the graph composed (TurboQuantKernels). Its own storage class, not a KVCache subclass; its row layouts are declared instate/layout. Only full-attention (non-rotating) layers use it; sliding-window layers stay bf16.Storage: 5 arrays per the vllm-metal reference's
turbo_quant_encodetuple — kIdx (int8 unpacked at kBits=8, else bit-packed uint8), kScales/kZeros (fp16, one pair per 32-group), vPacked (bit-packed uint8, no-op at vBits=8), vScales (fp16, one per 32-group). updateAndFetch quantizes ONLY the newly appended tokens (unlike QuantizedKVCache's whole-buffer-toQuantized conversion path — the streaming append is the whole point of a live cache) and returns the DEQUANTIZED bf16 active window so ops.sdpa runs unmodified — v1 pays a full-window dequant every step; the deferred-InvFWHT trick is a documented non-goal until the quality gate passes.Reads: each named read appends and attends with the kernel set the cache was built with (
turboQuantKernels): decoded values and the stock SDPA (denseTurboQuantKernels, Qwen 3.5's graph) or rotated values, the stock SDPA and the inverse rotation of its output (deferredTurboQuantKernels, Gemma 4's and MiniCPM5's). No caller sees rotated values. Decode passes no mask, a window the fused causal mask; at one query the two compute the same bits. A committed span is the window read: the graphs read TurboQuant storage the same way whether or not the positions are decided. This cache is full-attention only, so no read builds a mask array. Built empty, it encodes from token zero, unlike the served path'sBf16FirstTurboQuantKVCache, so its first chunk's numbers differ.Compiled decode: a concat slot. The stored prefix is decoded outside the recorded graph, in the domain the kernel set's read takes (
TurboQuantKernels.fetch), with the codec's own decode; inside it the new position is encoded, decoded from its codes and joined to the prefix, and the set reads the joined window as the ordinary decode read does. Decoding is per position and the codec's fused kernel computes the eager decode's arithmetic, so the joined window equals the window the ordinary read decodes. The codes leave the graph unpacked; the write after the step packs and stores them.