Class KvCacheService

constructor

• new KvCacheService(
      graph: CacheServiceGraph,
      parts?: KvCacheParts,
      settings?: KvCacheSettings,
  ): KvCacheService

  Parameters

  • graph: CacheServiceGraph
  • parts: KvCacheParts = {}
  • settings: KvCacheSettings = {}

  Returns KvCacheService

Readonlycodecs

checkpointEveryTokens

• get checkpointEveryTokens(): number | undefined

  Generation checkpoints every this many tokens; undefined: the service keeps none.

  Returns number | undefined

objects

• get objects(): ObjectCache<CheckpointAttachment[]> | undefined

  Exact tensor objects (encoder outputs) under the same budget and storage; absent without a prefix store.

  Returns ObjectCache<CheckpointAttachment[]> | undefined

onPut

• set onPut(observe: ((tokens: number[], namespace: string) => void) | null): void

  Observes every prefix the cache stores, with its tokens.

  Parameters

  • observe: ((tokens: number[], namespace: string) => void) | null

  Returns void

acquire

• acquire(request: StateRequest, namespace: string): AcquiredState

  request's state: its resumed generation, else the longest reusable prefix of its prompt in namespace (its adapters, prepared media, method), else fresh state. Ownership passes to the caller.

  Parameters

  • request: StateRequest
  • namespace: string

  Returns AcquiredState

admit

• admit(request: StateRequest): boolean

  Admit request beside the requests already holding state: true admits it (it holds its projected KV bytes until release), false means it waits. Throws for a request that can never be admitted: one past the caches' own position limit, or one over the KV budget alone.

  Parameters

  • request: StateRequest

  Returns boolean

boundary

• boundary(): void

  An execution boundary passed: the cache may give optional state back to the allocator. Never changes a running request.

  Returns void

checkpoint

• checkpoint(request: StateRequest, state: Omit<ContinuationState, "seed">): void

  Save a resumable point of request's generation (owned caches; the cache disposes them whatever happens).

  Parameters

  • request: StateRequest
  • state: Omit<ContinuationState, "seed">

  Returns void

close

• close(): Promise<DurabilityFlushResult>

  After every request drained: flush, then clear the state even when persistence fails (durable: false reports it). Idempotent.

  Returns Promise<DurabilityFlushResult>

closeSession

• closeSession(sessionId: string): void

  Drop a session's affinity without deleting reusable state.

  Parameters

  • sessionId: string

  Returns void

fits

• fits(request: StateRequest): boolean

  Whether admit would admit request now; holds nothing.

  Parameters

  • request: StateRequest

  Returns boolean

flush

• flush(): Promise<DurabilityFlushResult>

  Make every kept prefix and checkpoint durable, as far as storage allows.

  Returns Promise<DurabilityFlushResult>

keeps

• keeps(request: StateRequest): boolean

  Whether state request publishes may be kept: the scheduler skips copying state the cache would drop.

  Parameters

  • request: StateRequest

  Returns boolean

peekPrefixLen

• peekPrefixLen(promptIds: number[], namespace: string): number

  The longest prefix of promptIds held in RAM under namespace, which is looked up as given (not under the composition's paging).

  Parameters

  • promptIds: number[]
  • namespace: string

  Returns number

prefetch

• prefetch(
      promptIds: number[],
      namespace: string,
      sessionId?: string,
  ): Promise<() => void>

  Start reading promptIds' saved prefix in namespace before the request is admitted. The returned release ends the request's interest.

  Parameters

  • promptIds: number[]
  • namespace: string
  • OptionalsessionId: string

  Returns Promise<() => void>

publish

• publish(
      request: StateRequest,
      namespace: string,
      tokens: number[],
      state: PublishedState,
  ): void

  Offer state covering tokens for reuse in namespace; the cache keeps or drops it by its own policy. Ownership transfers on return; on a throw it stays with the caller.

  Parameters

  • request: StateRequest
  • namespace: string
  • tokens: number[]
  • state: PublishedState

  Returns void

release

• release(request: StateRequest, completed: boolean): void

  request is done: its KV bytes return to the budget; a completed generation's checkpoints are removed, an interrupted one keeps them.

  Parameters

  • request: StateRequest
  • completed: boolean

  Returns void

report

• report(): KvCacheReport

  Returns KvCacheReport

savedPrefixes

• savedPrefixes(): Iterable<readonly number[]>

  The token prefixes saved state holds (header metadata only).

  Returns Iterable<readonly number[]>

stats

• stats(): DurabilitySnapshotStats

  Pending persistence: prefix snapshots and spills, and checkpoint writes.

  Returns DurabilitySnapshotStats

stopIdleDemotion

• stopIdleDemotion(): void

  Stop idle demotion ahead of draining requests; the state stays.

  Returns void