mlx-bun library API
    Preparing search index...

    Interface LatentAttentionCache

    State for one multi-head latent attention layer (GLM-5.2's MLA): it stores the checkpoint-native compressed state per position (the latent and the decoupled RoPE key, and on a sparse-attention layer the DSA index key), never per-head keys and values. The attention block projects its queries and that state, appends the state here and reads back everything stored, from which it reconstructs or absorbs keys and values itself (the same split as GatedDeltaCache: the block keeps its projections, the cache keeps the state and its row layout). A layer composed with DSA appends an index key with every position and reads the index keys stored before its append to select positions.

    Rows: one row for a single sequence; a batch keeps its rows right-justified along the sequence axis, rowOffsets[b] positions in row b starting at leftPad[b]. Inputs to an append are borrowed and float32.

    interface LatentAttentionCache {
        batchSize: number | null;
        composed?: ComposedStorage;
        kvLoraRank: number;
        leftPad: readonly number[];
        minimumReusableOffset?: number;
        offset: number;
        prefillMaintenance?: {
            beginPrefill(): void;
            commitPrefill(rows: readonly number[]): void;
            endPrefill(): void;
        };
        ropeHeadDim: number;
        ropeOffsetArr?: MlxArray;
        rowOffsets: readonly number[];
        stateNeedsDispose?: boolean;
        appendLatent(
            latent: MlxArray,
            rope: MlxArray,
            indexKeys: MlxArray | null,
        ): LatentAttentionState;
        bytesPerToken?(): number;
        captureDonorAttention?(): KvDonorAttention;
        captureDonorRows?(): KvDonorRows;
        dispose(): void;
        isTrimmable(): boolean;
        maxAppendTokens?(): number;
        signature(): string;
        specRoundBegin?(): void;
        specRoundCommit?(): void;
        specRoundRollback?(keep: number): void;
        state(): MlxArray[];
        storedIndexKeys(): MlxArray | null;
        trim(n: number, bypass?: boolean): void;
    }

    Hierarchy (View Summary)

    • Cache
      • LatentAttentionCache

    Implemented by

    Index
    batchSize composed? kvLoraRank leftPad minimumReusableOffset? offset prefillMaintenance? ropeHeadDim ropeOffsetArr? rowOffsets stateNeedsDispose?
    appendLatent bytesPerToken? captureDonorAttention? captureDonorRows? dispose isTrimmable maxAppendTokens? signature specRoundBegin? specRoundCommit? specRoundRollback? state storedIndexKeys trim
    batchSize: number | null

    Rows in the layout, or null before the first append.

    composed?: ComposedStorage

    Present on a cache that composes the storage it holds (see ComposedStorage).

    kvLoraRank: number
    leftPad: readonly number[]
    minimumReusableOffset?: number

    Earliest prefix whose retained representation can resume generation. Irreversible transitions may preserve bytes while invalidating older precision boundaries. Reuse policy must not trim below this offset.

    offset: number
    prefillMaintenance?: {
        beginPrefill(): void;
        commitPrefill(rows: readonly number[]): void;
        endPrefill(): void;
    }

    A preparation method commits each request's own precision boundary. Splitting a forward for a sibling must not move that boundary.

    ropeHeadDim: number
    ropeOffsetArr?: MlxArray

    Compiled-decode trace adapters expose the RoPE offset as an int32 array input here; real caches leave it unset (static int path).

    rowOffsets: readonly number[]
    stateNeedsDispose?: boolean

    state() returned temporary views that the caller must release.

    • Physical cache storage consumed by one additional token for one row. Recurrent caches return 0 because their state does not grow with the sequence. Optional for stateless/training adapters that are never admitted or persisted.

      Returns number

    • Stable storage identity for compatibility guards and persistence. REQUIRED: every capability predicate (isPlainKvCache, isRotating*, packages/inference/src/state/persistence.ts codecs) is a string compare on this value, so a missing override used to fail OPEN into wrong routing (PRs #42/#43: BatchedRotatingCache shipped without one and batched joins dropped running rows). Wrappers report their own kind plus the inner kind.

      Returns string

    • Speculative verify-round support for NON-trimmable recurrent caches (SSMCache — gated-DeltaNet conv + recurrent state). A cache exposing these is spec-eligible without isTrimmable(): the serve loop arms a round before the verify forward (the layer snapshots pre-round state — free, MLX arrays are immutable — and records its position-local kernel inputs), then resolves it: commit on full accept, or rollback(keep) which restores the snapshot and bit-exactly REPLAYS the first keep window tokens through the recurrence (identical arithmetic prefix ⇒ identical state). Trimmable caches leave these unset and keep trim().

      Returns void

    • Drop the last n tokens (future writes overwrite them). bypass (default false → unchanged behavior) skips the isTrimmable guard for the speculative-decode rollback case (trim the last n ≤ γ tips right after a

      1 concat write); rotating caches then physically shrink the buffer tail.

      Parameters

      • n: number
      • Optionalbypass: boolean

      Returns void

    Settings
    Member Visibility

    On This Page
    Properties
    Methods