mlx-bun library API
    Preparing search index...

    Interface ModelHost

    Owns loaded models and their residency. Residency is by memory fit: a model that fits the budget loads beside the others; otherwise the least recently used unpinned, unleased model is evicted first. Eviction stops admitting to it, waits for in-flight work to finish (drain), flushes its KV and prefix state under the storage root, then releases it; acquiring it again resumes that state. The host never holds more than the budget, including during a swap. Load, unload and memory are published as events. Failures reject with a ModelHostError.

    interface ModelHost {
        policy: ResidencyPolicy;
        acquire(id: string, options?: AcquireOptions): Promise<ModelLease>;
        defaultFor(operation: ModelOperation): Promise<string | undefined>;
        pin(id: string): void;
        plan(id: string): Promise<ResidencyPlan>;
        resident(): readonly ResidentModel[];
        serve(id: string, options?: { signal?: AbortSignal }): Promise<void>;
        stats(id: string): ModelStats;
        unload(
            id: string,
            options?: { flush?: boolean; force?: boolean },
        ): Promise<void>;
        unpin(id: string): void;
    }

    Hierarchy (View Summary)

    Index
    policy
    acquire defaultFor pin plan resident serve stats unload unpin
    policy: ResidencyPolicy

    • Makes id the model defaultFor("generate") answers, loading it beside the resident ones when it fits, else in place of the least recently used unpinned one (its state is saved and resumes on return). Rejects with not-found for an id that is not a local model, not-switchable on a host that holds one model, and otherwise as acquire does.

      Parameters

      • id: string
      • Optionaloptions: { signal?: AbortSignal }

      Returns Promise<void>

    • Drains, optionally flushes state (default true), then releases. Rejects with in-use while a lease is held unless force; a model that is not resident resolves.

      Parameters

      • id: string
      • Optionaloptions: { flush?: boolean; force?: boolean }

      Returns Promise<void>

    Settings
    Member Visibility

    On This Page
    Properties
    Methods