mlx-bun library API
    Preparing search index...

    Interface CompositionRequest

    What the caller decided; each absent value takes the default named on it.

    interface CompositionRequest {
        draft?: { adaptive?: boolean; numDraftTokens: number } | null;
        grammarJump?: boolean;
        greedyDraft?: boolean;
        kv?: {
            fusedSdpa?: boolean;
            override?: KvQuantOverride;
            quantizedKvStart?: number;
            turboQuant?: TurboQuantScheme;
        };
        maxRows?: number;
        mixedPrefill?: boolean;
        mixedTokenBudget?: number;
        pagedKv?: { blockSize?: number; direct?: boolean }
        | null;
        prefillTailSplit?: boolean;
        vectorGreedySampling?: boolean;
    }
    Index
    draft? grammarJump? greedyDraft? kv? maxRows? mixedPrefill? mixedTokenBudget? pagedKv? prefillTailSplit? vectorGreedySampling?
    draft?: { adaptive?: boolean; numDraftTokens: number } | null

    The loaded drafter's tokens per round, and whether the scheduler adapts each round's count up to it. Absent or null: no drafter.

    grammarJump?: boolean

    Grammar-forced continuations enter the KV without one forward per token. Default false.

    greedyDraft?: boolean

    Speculative rows draft by argmax. Default false.

    kv?: {
        fusedSdpa?: boolean;
        override?: KvQuantOverride;
        quantizedKvStart?: number;
        turboQuant?: TurboQuantScheme;
    }

    The KV flags: --kv-quant (override), its TurboQuant spelling, --quantized-kv-start and --fused-sdpa (the affine read kernels: on, tiled-causal; off, unfused; absent, by the scheme's kind). Absent: bf16. A config override reads the artifact's kv_config.json from the model config.

    maxRows?: number

    The most rows one forward carries. Default DEFAULT_MAX_ROWS.

    mixedPrefill?: boolean

    One forward carries a decode group and a prefill group. Default false.

    mixedTokenBudget?: number

    Prompt tokens a mixed forward carries beside its decode rows. Default 256.

    pagedKv?: { blockSize?: number; direct?: boolean } | null

    Paged KV (--paged-kv): full-attention storage in pages of blockSize positions (default 256), read by the direct kernel where direct (default false; see PagedKvComposition). Absent or null: unpaged. Pages store what the KV scheme names for each layer.

    prefillTailSplit?: boolean

    A prompt's last token prefills alone, as the first one-position step. Default true.

    vectorGreedySampling?: boolean

    One argmax over a decode step whose live rows are all plain greedy. Default true.

    Settings
    Member Visibility

    On This Page
    Properties