mlx-bun library API
    Preparing search index...

    Interface PrefixCache<State, Attachment>

    Storage is independent of request scheduling and the method producing state. Both prefill and decode can publish processed-token prefixes. State is opaque here; its backend/method defines layout and alignment. Implementations own prefix selection, retention and storage tiers.

    interface PrefixCache<State, Attachment = never> {
        closeSession?(sessionId: string): void;
        prefetch?(
            prompt: number[],
            ns?: string,
            sessionId?: string,
        ): Promise<() => void>;
        put(
            tokens: number[],
            caches: State,
            ns?: string,
            retain?: () => void,
            attachments?: Attachment,
            sessionId?: string,
        ): void;
        reclaim?(): void;
        take(
            prompt: number[],
            ns?: string,
            sessionId?: string,
        ): PrefixCacheHit<State, Attachment> | null;
    }

    Type Parameters

    • State
    • Attachment = never

    Implemented by

    Index
    closeSession? prefetch? put reclaim? take

    • Prepare a reusable prefix before execution. The returned release ends request interest; cache residency and IO remain implementation-owned.

      Parameters

      • prompt: number[]
      • Optionalns: string
      • OptionalsessionId: string

      Returns Promise<() => void>

    • Transfer state and its optional backing release to storage. If this throws, ownership stays with the caller; successful return transfers ownership even when storage immediately spills or disposes the entry.

      Parameters

      • tokens: number[]
      • caches: State
      • Optionalns: string
      • Optionalretain: () => void
      • Optionalattachments: Attachment
      • OptionalsessionId: string

      Returns void

    Settings
    Member Visibility

    On This Page
    Methods