The layer's sliding window; null for full attention.
The per-forward masks of the model this cache belongs to.
The kernel this cache's reads attend with.
Real positions per row of a right-padded batch; null for unpadded rows.
ReadonlyattentionThe kernel this cache's reads attend with.
ReadonlymasksThe per-forward masks of the model this cache belongs to.
ReadonlyrowReal positions per row of a right-padded batch; null for unpadded rows.
ReadonlywindowThe layer's sliding window; null for full attention.
Decode: one new position per row (L = 1). Each row's query sees that
row's valid stored positions up to and including the one just appended,
limited to the cache's sliding window.
Window: L new positions per row, causal within the window, for prefill
chunks, prefill tails and speculative verify windows. Query i of a row
sees the row's earlier valid positions and window positions 0..i, limited
to the cache's sliding window. Left padding and prepared prefill padding are
never visible. Outputs at padded query positions are unspecified and the
caller ignores them. To append context without attending, make this call
and dispose the read unused.
Can trim(n) drop the last n tokens? (Ring caches lose trimability
once wrapped.)
Training-only adapter; never admitted, merged, or persisted.
Drop the last n tokens (future writes overwrite them). bypass (default
false → unchanged behavior) skips the isTrimmable guard for the
speculative-decode rollback case (trim the last n ≤ γ tips right after a
1 concat write); rotating caches then physically shrink the buffer tail.
Stateless cache for the training and scoring forward. Training is always a single offset-0 full-sequence pass, so nothing is stored and every append reads exactly the keys and values it is handed; the offset stays 0. The statelessness is REQUIRED: gradient checkpointing re-runs a layer's closure during the backward recompute, and a layer that shares another layer's keys and values reads them through that layer's cache again, so an append must leave the cache as it found it.
Reads: pass-through views of
kandv, attended withattention. WithoutrowLengths: no mask for decode, the fused causal mask for a window, and the offset-0 sliding-window mask once a window is longer thanwindow. WithrowLengths(a right-padded batch whose rows hold that many real positions), both reads take the padded-batch mask (paddedTrainingMask) for the cache's window, so no real query sees a padded key. A layer with a sliding window gets a cache composed with it; the caches of one forward sharemasks, so its layers build each mask once.