Tier policy: a pending write is not a lower-tier copy. This controls eviction only; persistence completion does not itself remove an entry.
ReadonlymaxObserver after publication, independent of persistence policy.
The backend measures pressure; this store chooses which optional snapshots to release. No storage I/O runs at a reclaim boundary.
ProtectedpromoteDrop session affinity without deleting reusable checkpoints.
Evict entries older than idleMs. Tiered residency policy first ensures SSD coverage. No-op without a cold tier. Returns entries removed from RAM.
Read-only lookup of an immutable, cache-owned checkpoint.
Longest usable prefix length available for prompt WITHOUT taking
(the SSD tier compares this against its own best before restoring).
Start cold reads during request preparation, coalescing identical IO. Active request interest keeps a just-restored donor resident until the caller has consumed it. Cancellation releases interest through the lease.
OptionalsessionId: stringInsert (or reinsert) an entry; evicts LRU entries over the byte cap (spilling each to the cold tier first, when one is attached). If the entry itself exceeds the cap it is spilled + disposed, not stored.
Optionalretain: () => voidOptionalattachments: CheckpointAttachment[]OptionalsessionId: stringGive optional retained state back to the allocator at an execution boundary. This never rejects or changes the active request.
Release LRU entries until the caller's live-memory check is satisfied. Unlike background eviction, pressure relief must finish the SSD write and release the buffers now, not enqueue clones that keep them alive. Call under the engine lock, after taking the request's reusable prefix.
Find the entry with the longest usable common prefix of prompt
across BOTH tiers. Every RAM entry lends zero-copy views and retains its
donor, including exact-boundary recurrent state and method attachments.
The caller owns the returned views —
dispose them or put() an extended entry, honoring retain either way.
Usable prefix = common prefix capped at prompt.length - 1 (at least one token must be forwarded to produce logits). Entries longer than the prefix need cache.trim(); ring caches lose trimability once wrapped — those entries only match in full.
Tier order: the cold tier wins only with a STRICTLY longer usable prefix (restore reads storage and materializes native arrays). A cold entry with an untrimmable divergent tail (ring post-wrap, SSM) is dropped and the RAM candidate (or a fresh prefill) serves. hits/misses count RAM candidacy only (cold restores are counted by the tier itself), preserving the /stats meaning.
OptionalsessionId: string
Storage is independent of request scheduling and the method producing state. Both prefill and decode can publish processed-token prefixes. State is opaque here; its backend/method defines layout and alignment. Implementations own prefix selection, retention and storage tiers.