Interface KvCacheParts

The parts a service composes. createKvCacheService builds them from the serve flags; CPU tests and replacement runtimes may supply their own.

interface KvCacheParts {
    continuations?:
        | {
            everyTokens: number;
            persistence: ContinuationPersistence;
            store: ContinuationStore;
        }
        | null;
    prefixes?: PromptCache
    | null;
    saved?: SsdCacheStore | null;
}

Properties

continuations?:
    | {
        everyTokens: number;
        persistence: ContinuationPersistence;
        store: ContinuationStore;
    }
    | null

Generation checkpoints, written every everyTokens generated tokens.

prefixes?: PromptCache | null

Reusable prefixes, tiered over saved state when there is any. Null: the service reuses nothing and keeps nothing.

saved?: SsdCacheStore | null

Saved state the prefixes tier over; reported, never reached by callers.