mlx-bun library API
    Preparing search index...

    Interface BatchableCache

    Capability contract for cache layouts that own their dynamic-row batching. The scheduler uses this for every storage family (plain and quantized KV, sliding windows, recurrent state, and layouts such as GLM's compressed MLA/DSA state) instead of teaching the scheduler their tensor layout. mergeRows writes into an empty cache and does not consume inputs: its sources are a running layout or one serial cache, then one serial cache per joining row. Extracted rows are independent owned serial caches.

    interface BatchableCache {
        batchSize: number | null;
        composed?: ComposedStorage;
        leftPad: readonly number[];
        maxTokens?: number;
        minimumReusableOffset?: number;
        offset: number;
        prefillMaintenance?: {
            beginPrefill(): void;
            commitPrefill(rows: readonly number[]): void;
            endPrefill(): void;
        };
        ropeOffsetArr?: MlxArray;
        rowOffsets: readonly number[];
        stateNeedsDispose?: boolean;
        bytesPerToken?(): number;
        canPublishRow?(row: number, tokens: number): boolean;
        captureDonorAttention?(): KvDonorAttention;
        captureDonorRows?(): KvDonorRows;
        dispose(): void;
        extractRow(row: number): Cache;
        filterRows(keep: readonly number[]): void;
        isTrimmable(): boolean;
        makeEmptyBatch(): BatchableCache;
        maxAppendTokens?(): number;
        mergeRows(rows: readonly Cache[]): void;
        projectedBytes(tokens: number): number;
        signature(): string;
        specRoundBegin?(): void;
        specRoundCommit?(): void;
        specRoundRollback?(keep: number): void;
        state(): MlxArray[];
        trim(n: number, bypass?: boolean): void;
    }

    Hierarchy (View Summary)

    Implemented by

    Index
    batchSize composed? leftPad maxTokens? minimumReusableOffset? offset prefillMaintenance? ropeOffsetArr? rowOffsets stateNeedsDispose?
    bytesPerToken? canPublishRow? captureDonorAttention? captureDonorRows? dispose extractRow filterRows isTrimmable makeEmptyBatch maxAppendTokens? mergeRows projectedBytes signature specRoundBegin? specRoundCommit? specRoundRollback? state trim
    batchSize: number | null
    composed?: ComposedStorage

    Present on a cache that composes the storage it holds (see ComposedStorage).

    leftPad: readonly number[]
    maxTokens?: number
    minimumReusableOffset?: number

    Earliest prefix whose retained representation can resume generation. Irreversible transitions may preserve bytes while invalidating older precision boundaries. Reuse policy must not trim below this offset.

    offset: number
    prefillMaintenance?: {
        beginPrefill(): void;
        commitPrefill(rows: readonly number[]): void;
        endPrefill(): void;
    }

    A preparation method commits each request's own precision boundary. Splitting a forward for a sibling must not move that boundary.

    ropeOffsetArr?: MlxArray

    Compiled-decode trace adapters expose the RoPE offset as an int32 array input here; real caches leave it unset (static int path).

    rowOffsets: readonly number[]
    stateNeedsDispose?: boolean

    state() returned temporary views that the caller must release.

    • Physical cache storage consumed by one additional token for one row. Recurrent caches return 0 because their state does not grow with the sequence. Optional for stateless/training adapters that are never admitted or persisted.

      Returns number

    • Whether row can be published as a cache covering exactly tokens positions. State that cannot be trimmed (recurrent) refuses a row whose own count differs: an entry keyed to other tokens would corrupt every later exact hit. Absent: any row can be published.

      Parameters

      • row: number
      • tokens: number

      Returns boolean

    • Stable storage identity for compatibility guards and persistence. REQUIRED: every capability predicate (isPlainKvCache, isRotating*, packages/inference/src/state/persistence.ts codecs) is a string compare on this value, so a missing override used to fail OPEN into wrong routing (PRs #42/#43: BatchedRotatingCache shipped without one and batched joins dropped running rows). Wrappers report their own kind plus the inner kind.

      Returns string

    • Speculative verify-round support for NON-trimmable recurrent caches (SSMCache — gated-DeltaNet conv + recurrent state). A cache exposing these is spec-eligible without isTrimmable(): the serve loop arms a round before the verify forward (the layer snapshots pre-round state — free, MLX arrays are immutable — and records its position-local kernel inputs), then resolves it: commit on full accept, or rollback(keep) which restores the snapshot and bit-exactly REPLAYS the first keep window tokens through the recurrence (identical arithmetic prefix ⇒ identical state). Trimmable caches leave these unset and keep trim().

      Returns void

    • Drop the last n tokens (future writes overwrite them). bypass (default false → unchanged behavior) skips the isTrimmable guard for the speculative-decode rollback case (trim the last n ≤ γ tips right after a

      1 concat write); rotating caches then physically shrink the buffer tail.

      Parameters

      • n: number
      • Optionalbypass: boolean

      Returns void

    Settings
    Member Visibility

    On This Page
    Properties
    Methods