Optional ReadonlycomposedPresent on a cache that composes the storage it holds (see ComposedStorage).
OptionalminimumEarliest prefix whose retained representation can resume generation. Irreversible transitions may preserve bytes while invalidating older precision boundaries. Reuse policy must not trim below this offset.
Optional ReadonlyprefillA preparation method commits each request's own precision boundary. Splitting a forward for a sibling must not move that boundary.
Optional ReadonlyropeCompiled-decode trace adapters expose the RoPE offset as an int32 array input here; real caches leave it unset (static int path).
Optional Readonlystatestate() returned temporary views that the caller must release.
Decode: one new position per row (L = 1). Each row's query sees that
row's valid stored positions up to and including the one just appended,
limited to the cache's sliding window.
Window: L new positions per row, causal within the window, for prefill
chunks, prefill tails and speculative verify windows. Query i of a row
sees the row's earlier valid positions and window positions 0..i, limited
to the cache's sliding window. Left padding and prepared prefill padding are
never visible. Outputs at padded query positions are unspecified and the
caller ignores them. To append context without attending, make this call
and dispose the read unused.
OptionalbytesPhysical cache storage consumed by one additional token for one row. Recurrent caches return 0 because their state does not grow with the sequence. Optional for stateless/training adapters that are never admitted or persisted.
OptionalcaptureOptionalcaptureCan trim(n) drop the last n tokens? (Ring caches lose trimability
once wrapped.)
OptionalmaxMaximum committed positions before this state changes precision.
Stable storage identity for compatibility guards and persistence. REQUIRED: every capability predicate (isPlainKvCache, isRotating*, packages/inference/src/state/persistence.ts codecs) is a string compare on this value, so a missing override used to fail OPEN into wrong routing (PRs #42/#43: BatchedRotatingCache shipped without one and batched joins dropped running rows). Wrappers report their own kind plus the inner kind.
OptionalspecSpeculative verify-round support for NON-trimmable recurrent caches
(SSMCache — gated-DeltaNet conv + recurrent state). A cache exposing
these is spec-eligible without isTrimmable(): the serve loop arms a
round before the verify forward (the layer snapshots pre-round state —
free, MLX arrays are immutable — and records its position-local kernel
inputs), then resolves it: commit on full accept, or rollback(keep)
which restores the snapshot and bit-exactly REPLAYS the first keep
window tokens through the recurrence (identical arithmetic prefix ⇒
identical state). Trimmable caches leave these unset and keep trim().
OptionalspecOptionalspecDrop the last n tokens (future writes overwrite them). bypass (default
false → unchanged behavior) skips the isTrimmable guard for the
speculative-decode rollback case (trim the last n ≤ γ tips right after a
1 concat write); rotating caches then physically shrink the buffer tail.
Optionalbypass: boolean
A key/value cache that owns its attention read. Each method appends the new keys and values for the phase it names and returns the read for that phase. The caller never sees stored keys or values and never chooses a kernel. The storage (bf16, affine, TurboQuant, paged, rotating, delayed quantization) is not visible here: the loader composes the cache, and the cache fixes one kernel per read at construction.
Every append has these obligations:
k[B, Hkv, L, Dk] andv[B, Hkv, L, Dv] are borrowed. The caller disposes them after the call returns.kalready carries the positional encoding for the positions it occupies, which start at each row's offset.rowOffsets,leftPadand padding prepared throughPaddedPrefillCachefor a batchable layout. Each row appends at its own offset, and the offsets have advanced byLwhen the call returns.L, from the query shape or from a mask. Calling a method outside its phase gives an undefined result.