mlx-bun library API
    Preparing search index...

    Interface SpeculativeRowLayout

    Verification rows: the row layout a speculative method keeps its target rows in, holding the speculative transaction. Every storage a cache lego holds names one (state's targetCacheLayout), so any lego serves speculation.

    The method arms a round before the verify window (specRoundBegin), appends the window, then resolves it: specRoundCommit keeps every row's window; specRoundRollback(keep) keeps the first keep positions of each row's window (one count for every row, or one per row). After either, every row is exactly as if only its kept positions had been appended: the same offsets and stored positions, the same reads from then on.

    interface SpeculativeRowLayout {
        batchSize: number | null;
        composed?: ComposedStorage;
        leftPad: readonly number[];
        maxTokens?: number;
        minimumReusableOffset?: number;
        offset: number;
        prefillMaintenance?: {
            beginPrefill(): void;
            commitPrefill(rows: readonly number[]): void;
            endPrefill(): void;
        };
        ropeOffsetArr?: MlxArray;
        rowOffsets: readonly number[];
        stateNeedsDispose?: boolean;
        bytesPerToken?(): number;
        canPublishRow?(row: number, tokens: number): boolean;
        captureDonorAttention?(): KvDonorAttention;
        captureDonorRows?(): KvDonorRows;
        dispose(): void;
        extractRow(row: number): Cache;
        filterRows(keep: readonly number[]): void;
        isTrimmable(): boolean;
        makeEmptyBatch(): SpeculativeRowLayout;
        maxAppendTokens?(): number;
        mergeRows(rows: readonly Cache[]): void;
        projectedBytes(tokens: number): number;
        signature(): string;
        specRoundBegin(): void;
        specRoundCommit(): void;
        specRoundRollback(keep: number | readonly number[]): void;
        state(): MlxArray[];
        trim(n: number, bypass?: boolean): void;
    }

    Hierarchy (View Summary)

    Implemented by

    Index
    batchSize composed? leftPad maxTokens? minimumReusableOffset? offset prefillMaintenance? ropeOffsetArr? rowOffsets stateNeedsDispose?
    bytesPerToken? canPublishRow? captureDonorAttention? captureDonorRows? dispose extractRow filterRows isTrimmable makeEmptyBatch maxAppendTokens? mergeRows projectedBytes signature specRoundBegin specRoundCommit specRoundRollback state trim
    batchSize: number | null
    composed?: ComposedStorage

    Present on a cache that composes the storage it holds (see ComposedStorage).

    leftPad: readonly number[]
    maxTokens?: number
    minimumReusableOffset?: number

    Earliest prefix whose retained representation can resume generation. Irreversible transitions may preserve bytes while invalidating older precision boundaries. Reuse policy must not trim below this offset.

    offset: number
    prefillMaintenance?: {
        beginPrefill(): void;
        commitPrefill(rows: readonly number[]): void;
        endPrefill(): void;
    }

    A preparation method commits each request's own precision boundary. Splitting a forward for a sibling must not move that boundary.

    ropeOffsetArr?: MlxArray

    Compiled-decode trace adapters expose the RoPE offset as an int32 array input here; real caches leave it unset (static int path).

    rowOffsets: readonly number[]
    stateNeedsDispose?: boolean

    state() returned temporary views that the caller must release.

    • Whether row can be published as a cache covering exactly tokens positions. State that cannot be trimmed (recurrent) refuses a row whose own count differs: an entry keyed to other tokens would corrupt every later exact hit. Absent: any row can be published.

      Parameters

      • row: number
      • tokens: number

      Returns boolean

    • Stable storage identity for compatibility guards and persistence. REQUIRED: every capability predicate (isPlainKvCache, isRotating*, packages/inference/src/state/persistence.ts codecs) is a string compare on this value, so a missing override used to fail OPEN into wrong routing (PRs #42/#43: BatchedRotatingCache shipped without one and batched joins dropped running rows). Wrappers report their own kind plus the inner kind.

      Returns string

    • Speculative verify-round support for NON-trimmable recurrent caches (SSMCache — gated-DeltaNet conv + recurrent state). A cache exposing these is spec-eligible without isTrimmable(): the serve loop arms a round before the verify forward (the layer snapshots pre-round state — free, MLX arrays are immutable — and records its position-local kernel inputs), then resolves it: commit on full accept, or rollback(keep) which restores the snapshot and bit-exactly REPLAYS the first keep window tokens through the recurrence (identical arithmetic prefix ⇒ identical state). Trimmable caches leave these unset and keep trim().

      Returns void

    • Drop the last n tokens (future writes overwrite them). bypass (default false → unchanged behavior) skips the isTrimmable guard for the speculative-decode rollback case (trim the last n ≤ γ tips right after a

      1 concat write); rotating caches then physically shrink the buffer tail.

      Parameters

      • n: number
      • Optionalbypass: boolean

      Returns void

    Settings
    Member Visibility

    On This Page
    Properties
    Methods