mlx-bun library API
    Preparing search index...

    Interface CompiledDecodeCache

    A cache the compiled decode step can drive: it declares the slot it occupies in the closure for the coming step and applies the closure's outputs back.

    interface CompiledDecodeCache {
        composed?: ComposedStorage;
        minimumReusableOffset?: number;
        offset: number;
        prefillMaintenance?: {
            beginPrefill(): void;
            commitPrefill(rows: readonly number[]): void;
            endPrefill(): void;
        };
        ropeOffsetArr?: MlxArray;
        stateNeedsDispose?: boolean;
        bytesPerToken?(): number;
        captureDonorAttention?(): KvDonorAttention;
        captureDonorRows?(): KvDonorRows;
        commitDecodeStep(slot: DecodeSlot, outputs: MlxArray[]): MlxArray[];
        decodeInputs(plan: DecodeStepPlan, step: DecodeStepInputs): MlxArray[];
        decodePhase(): "concat" | "ring";
        decodeSlot(plan: DecodeStepPlan): DecodeSlot;
        dispose(): void;
        isTrimmable(): boolean;
        maxAppendTokens?(): number;
        prepareDecodeStep(): DecodeStepPlan;
        signature(): string;
        specRoundBegin?(): void;
        specRoundCommit?(): void;
        specRoundRollback?(keep: number): void;
        state(): MlxArray[];
        trim(n: number, bypass?: boolean): void;
    }

    Hierarchy (View Summary)

    • Cache
      • CompiledDecodeCache

    Implemented by

    Index
    composed? minimumReusableOffset? offset prefillMaintenance? ropeOffsetArr? stateNeedsDispose?
    bytesPerToken? captureDonorAttention? captureDonorRows? commitDecodeStep decodeInputs decodePhase decodeSlot dispose isTrimmable maxAppendTokens? prepareDecodeStep signature specRoundBegin? specRoundCommit? specRoundRollback? state trim
    composed?: ComposedStorage

    Present on a cache that composes the storage it holds (see ComposedStorage).

    minimumReusableOffset?: number

    Earliest prefix whose retained representation can resume generation. Irreversible transitions may preserve bytes while invalidating older precision boundaries. Reuse policy must not trim below this offset.

    offset: number
    prefillMaintenance?: {
        beginPrefill(): void;
        commitPrefill(rows: readonly number[]): void;
        endPrefill(): void;
    }

    A preparation method commits each request's own precision boundary. Splitting a forward for a sibling must not move that boundary.

    ropeOffsetArr?: MlxArray

    Compiled-decode trace adapters expose the RoPE offset as an int32 array input here; real caches leave it unset (static int path).

    stateNeedsDispose?: boolean

    state() returned temporary views that the caller must release.

    • Physical cache storage consumed by one additional token for one row. Recurrent caches return 0 because their state does not grow with the sequence. Optional for stateless/training adapters that are never admitted or persisted.

      Returns number

    • Stable storage identity for compatibility guards and persistence. REQUIRED: every capability predicate (isPlainKvCache, isRotating*, packages/inference/src/state/persistence.ts codecs) is a string compare on this value, so a missing override used to fail OPEN into wrong routing (PRs #42/#43: BatchedRotatingCache shipped without one and batched joins dropped running rows). Wrappers report their own kind plus the inner kind.

      Returns string

    • Speculative verify-round support for NON-trimmable recurrent caches (SSMCache — gated-DeltaNet conv + recurrent state). A cache exposing these is spec-eligible without isTrimmable(): the serve loop arms a round before the verify forward (the layer snapshots pre-round state — free, MLX arrays are immutable — and records its position-local kernel inputs), then resolves it: commit on full accept, or rollback(keep) which restores the snapshot and bit-exactly REPLAYS the first keep window tokens through the recurrence (identical arithmetic prefix ⇒ identical state). Trimmable caches leave these unset and keep trim().

      Returns void

    • Drop the last n tokens (future writes overwrite them). bypass (default false → unchanged behavior) skips the isTrimmable guard for the speculative-decode rollback case (trim the last n ≤ γ tips right after a

      1 concat write); rotating caches then physically shrink the buffer tail.

      Parameters

      • n: number
      • Optionalbypass: boolean

      Returns void

    Settings
    Member Visibility

    On This Page
    Properties
    Methods