Interface StateRequest

One request as the cache sees it: fields of the request the scheduler admits. The request object identifies it from admit to release.

interface StateRequest {
    cacheSessionId?: string;
    continuation?: ContinuationRequest;
    maxTokens: number;
    promptIds: number[];
    reusesPrefix?: boolean;
}

Properties

cacheSessionId?: string

A replayable generation: the service checkpoints it and resumes it.

maxTokens: number

Tokens the request may generate: with the prompt, its KV projection and page capacity.

promptIds: number[]

reusesPrefix?: boolean

False for a request whose prefix has no identity (prepared media without one): it takes fresh state, reuses none, and its state is not kept.