Server API

mlx-bun serve exposes an OpenAI-compatible HTTP API on one model. The request’s model field is ignored; the loaded model’s id is echoed back. The default concurrency cap is eight. Eligible requests use continuous batching; --batch 1 pins serial execution. Both use the shared generation session for output, cancellation and cleanup. Each model implementation supplies its own execution methods; protocol handlers do not select a concrete model class. Embedding execution also acquires the shared model/GPU lease, so it waits for active generation or model mutation before running.

This is the canonical wire reference: every route the server registers is listed below, and every field documented here exists in the code that serves it (src/server.ts, src/serve/*-routes.ts, src/serve/isolate.ts, src/pi-web.ts). Start flags (--port, --memory-budget, --prompt-cache, --batch, --kv-quant, --thinking, sampling defaults) and which combinations compose live in server-config.md; the CLI in cli.md.

Route index

Dispatch order matters only for /ws/chat (upgrade) and the static shell; every other path is unique. Unmatched paths return 404 {"error":{"message":"not found"}}.

Method	Path	Section
POST	/v1/chat/completions	Chat completions
POST	/v1/completions	Text completions
POST	/v1/messages	Anthropic Messages
POST	/v1/responses	OpenAI Responses
POST	/v1/embeddings	Embeddings
POST	/v1/audio/transcriptions, /v1/audio/translations	Audio transcription
POST	/v1/audio/sessions, /v1/audio/sessions/<id>/audio, /v1/audio/sessions/<id>/finish	Streaming sessions
DELETE	/v1/audio/sessions/<id>	Streaming sessions
POST	/admin/transcription/unload	Audio transcription
GET	/v1	API index
GET	/v1/models, /v1/models/<id>	Models
GET / POST	/v1/adapters	Adapters
GET	/v1/adapters/available	Adapters
DELETE	/v1/adapters/<id>	Adapters
GET	/health	Health
GET	/stats	Stats
GET	/fit	Fit
GET	/library	Library
GET	/downloads	Downloads
POST	/admin/cache/session/close	Close cache session
POST	/admin/cache/flush	SSD cache flush
GET	/api/jobs, /api/jobs/<id>, /api/jobs/<id>/stream	Jobs
POST	/api/finetune/submit, /inspect-dataset, /merge, /export	Fine-tune
POST	/api/quantize/inspect, /submit, /resolve-folder	Quantize
GET / POST	/api/dataset/templates, /api/dataset/submit	Dataset
POST	/api/model/resolve-folder	Quantize
POST	/api/{quantize,finetune,dataset}/push	Push to Hub
GET / POST	/api/gc/plan, /api/gc/execute	GC
GET / POST	/api/memory/*	Memory
GET	/v1/memory/synthesize	Memory synthesis
GET / POST / DELETE	/api/settings/hf-token, /api/settings/tool-approvals	Settings
GET / POST	/api/hub/local, /search, /download, /serve	Model Hub
GET	/api/sessions/search, /api/sessions/export	Sessions
GET	/, /assets/*, /manifest.webmanifest, /sw.js, /curves, /curve-terrain, /dag	Static
GET	/status, /chat, /quantize, /finetune, /dataset	302 to /#<path> (Static)
POST	/signal, /generate	Curve Designer
WS	/ws/chat	Web chat WebSocket

Internal routes (not part of the client contract):

Method	Path	Where	Notes
POST	/admin/drain	src/server.ts	Only registered when the server is bound to a unix socket (--unix, i.e. an --isolate engine child). Quiesces the gateway and demotes the whole prompt cache to the SSD tier; returns { "drained": true, "demotions": N }. Never exposed on TCP.
POST	/admin/lease	src/server.ts	Unix-socket workers only. Flushes durability, then holds the gateway’s native execution lease while the response connection stays open. The isolation parent uses it for managed GPU jobs; disconnect releases the lease. Never exposed on TCP.
GET	/engine	src/serve/isolate.ts	Only on the --isolate parent proxy: { isolated: true, pid, restarts, socket, response_store, pool?: { resident, default } }. Worker fields describe the current default worker, or are null while it is evicted; inspection never loads it.

Under --isolate, the parent proxies everything to the engine child; /ws/chat is answered 501 there (web chat needs a non-isolated server), /v1/chat/completions, /v1/completions, /v1/messages, /v1/responses, and /v1/embeddings are routed by the body’s model field when a --model-pool is configured, an unreachable child is 502 {"error":{"type":"engine_unavailable"}}, and a client disconnect mid-proxy is 499.

Diagnostics: x-mlx-bun-trace-id and MLX_BUN_P2R_TRACE

/v1/chat/completions and /v1/completions read an optional x-mlx-bun-trace-id request header. With MLX_BUN_P2R_TRACE=1 set on the server, each of those requests emits one JSON record to stderr, prefixed [p2r] , when it finishes (src/serve/prompt-response-trace.ts). Without the env var nothing is allocated and the header is ignored.

{
  "version": 1,
  "traceId": "<x-mlx-bun-trace-id, or the request id when absent>",
  "requestId": "chatcmpl-…" | "cmpl-…",
  "route": "/v1/chat/completions" | "/v1/completions",
  "clock": "monotonic-ms",
  "outcome": "success" | "error" | "abort",
  "totalMs": 0.0,
  "events": [                      // sorted by startMs
    { "phase": "request.body_parse", "startMs": 0.0, "durationMs": 0.0,
      "attributes": { /* optional string|number|boolean|null map */ } }
  ],
  "attributes": { "stage": "body_parse" | "prepare_completion" }  // on some errors
}

Phases: request.body_parse, request.prompt_prepare, completion.total, completion.placement, engine.admission_wait, cache.lookup_restore, prefill.total, prefill.batch_setup, prefill.chunk, prefill.kv_maintenance, token_zero.total, token_zero.forward, token_zero.head, token_zero.sample, response.first_write, response.final_write. Marks (response.first_write, response.final_write) have durationMs: 0.

POST /v1/chat/completions

Request body (OpenAI chat schema; unknown fields ignored):

{
  "messages": [ /* role: system | user | assistant | tool */ ],
  "stream": false,
  "max_tokens": 1024,            // or max_completion_tokens (wins); unset =
                                 // no cap — generate until EOS or the
                                 // an explicit operator limit is reached;
                                 // --max-tokens sets a server default
  "temperature": 0.7,            // 0 = greedy
  "top_p": 0, "top_k": 0,        // 0 = off
  "seed": 1234,                  // omit for time-derived
  "min_p": 0.05,                 // min-p sampling (0 = off)
  "xtc_probability": 0.5,        // XTC sampling: chance per step of dropping
  "xtc_threshold": 0.1,          //   every token above threshold except the
                                 //   least likely (EOS + newline exempt,
                                 //   matching mlx_lm.server)
  "logit_bias": { "42": -5.0 },  // additive per-token-id logit bias
  "repetition_penalty": 1.1,     // optional
  "repetition_context_size": 20, // recent-token window (0 = whole history)
  "presence_penalty": 0.0,       // subtracted once if the token occurred
  "presence_context_size": 20,   //   in the window (mlx-lm extension)
  "frequency_penalty": 0.0,      // subtracted per occurrence in the window
  "frequency_context_size": 20,  //   (mlx-lm extension)
  "logprobs": false,             // BOOL (mlx_lm.server's type, even on
                                 //   /v1/completions): return each emitted
                                 //   token's logprob
  "top_logprobs": -1,            // int in [0, 11] or -1 = unset (mlx-lm's
                                 //   validation — its cap is 11, not
                                 //   OpenAI's 20); k > 0 returns the top-k
                                 //   (token, logprob) pairs per position
  "stop": "\n\n",                // or ["###", "\n\n"] (spec: up to 4)
  "tools": [ /* OpenAI function tools */ ],
  "tool_choice": "auto",         // "none" disables tools; "auto" and the
                                 //   {type:"function", function:{name}}
                                 //   object are accepted and treated as auto
  "chat_template_kwargs": {      // forwarded to the chat template
    "enable_thinking": false,    // MiniCPM5 / Qwen3.5/3.8 / GLM-5.2: <think>
                                 // channel on/off (Qwen3.8 default: ON)
    "preserve_thinking": true    // Qwen3.8: keep think blocks from earlier
                                 // assistant turns in the rendered prompt
                                 // (template default true — better prompt-
                                 // cache reuse in agent loops). Ignored by
                                 // templates that don't read it.
  },
  "reasoning_effort": "medium",  // "none"|"minimal"|"low"|"medium"|"high"|"xhigh"
                                 // gates enable_thinking on Qwen3.5/3.8/
                                 // MiniCPM5: "none" → off, any other level →
                                 // on. Only consulted when chat_template_
                                 // kwargs.enable_thinking is not explicitly
                                 // set — an explicit enable_thinking always
                                 // wins. On templates with a reasoning-depth
                                 // variable (Qwen3.8) the level also maps
                                 // into the template: minimal/low → low,
                                 // medium → medium, high/xhigh → xhigh.
                                 // Any OTHER string is a 400.
  "hlg": {                       // HLG tone-curve sampling (per request).
    "enabled": true,             // merged over --hlg-sampling server defaults.
    "width": 4,                  // logit-width of the tone plateau
    "shoulder": 4,               // top shoulder width
    "toe": 6,                    // bottom toe width
    "pivot_offset": 6            // pivot point offset from top
  },
  "response_format":             // structured output (grammar-constrained
    { "type": "json_object" },   //   decoding) — also guided_grammar,
                                 //   guided_regex, guided_choice,
                                 //   structured_outputs; see "Structured
                                 //   output" below
  "adapter": "id"                // LoRA: "id", stacked "a+b", or "none"
}

Sampling defaults follow the model author’s generation_config.json when a field is omitted (optiq serve’s gen_config behavior); explicit request values always win. MiniCPM5 defaults to the no-think direct answer mode unless chat_template_kwargs.enable_thinking is true.

Assistant history accepts reasoning as an alias for reasoning_content, including Pi’s reasoning on tool-call messages. An explicit reasoning_content string takes precedence, even when empty. Normalization preserves the original message and exposes the canonical field to templates, so Qwen’s preserve_thinking: true retains that history across tool turns.

GLM-5.2 also defaults to no-think: its generation primer is <think></think>. Enabling thinking renders Reasoning Effort: Max in the system turn and leaves an open <think> primer for generation. The same precedence applies through Chat Completions, Anthropic Messages, and Responses because all three normalize into this one chat request path.

stop sequences are matched on decoded text, not token ids, so a sequence that spans token boundaries still fires. Generation halts at the first match; the stop sequence itself is excluded from the content and finish_reason is "stop".

Message content is a string or an array of parts: { "type": "text", "text": ... } and { "type": "image_url", "image_url": { "url": "data:image/png;base64,..." } } ({ "type": "image", ... } is accepted as an alias; http/https URLs also accepted; PNG, JPEG, HEIC, AVIF, WebP, TIFF, GIF, BMP via native OS codecs; requires a model with the vision sidecar).

Remote (http/https) media URLs — image, audio, and video alike — go through a destination policy (src/media-fetch.ts): private/loopback/ link-local hosts are refused by default (the request URL is attacker-controlled input — SSRF), every redirect hop is re-validated, and the fetch has a 10 s timeout and a 64 MB response cap (video: 256 MB — clips are legitimately larger; frame sampling truncates long clips server-side). Violations are clean 400s (prompt build failed: image url rejected: …). Serving media from a LAN host (a NAS, another machine) is the opt-in --allow-private-media flag (or MLX_BUN_ALLOW_PRIVATE_MEDIA=1). data: URLs are decoded locally and never policy-checked.

Video input (Qwen3.5-family models)

Video rides the same content-part array:

{ "type": "video_url", "video_url": { "url": "data:video/mp4;base64,…|http(s)://…" } }
{ "type": "video",     "data": "<base64>" }                          // alias

The container decodes through macOS’s own AVFoundation codecs (mp4/mov, H.264/HEVC/ProRes/… — whatever this macOS plays) via the mlx-bun-frame-extract helper that ships beside the binary and in the native pack; from a source tree it compiles on first use with the Xcode CLT. Frames are sampled at 2 fps, capped at 768 frames (src/vision/video-frames.ts), then run the gated Qwen3-VL video pipeline (temporal frame pairs, per-frame-group attention, 3D mRoPE positions). Qwen3.5-family models only; video never combines with audio parts (400 "video and audio content parts cannot be combined"), and a non-Qwen model answers video parts with a 400 naming the model.

Audio input

Audio rides the same content-part array (OpenAI-canonical shape plus two aliases):

{ "type": "input_audio", "input_audio": { "data": "<base64>", "format": "wav" } }
{ "type": "audio",       "data": "<base64>" }                        // alias
{ "type": "audio_url",   "audio_url": { "url": "data:…|http(s)://…" } } // alias

(audio_url http/https fetches follow the same destination policy, timeout, and size cap as image_url — see above.)

curl -s http://localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -d "$(jq -n --arg b64 "$(base64 -i clip.wav)" '{
    messages: [{ role: "user", content: [
      { type: "input_audio", input_audio: { data: $b64, format: "wav" } },
      { type: "text", text: "Transcribe this audio." }
    ]}],
    max_tokens: 64, temperature: 0
  }')"

• Models: any model whose config.json carries an audio_config AND whose optiq_vision.safetensors sidecar ships the audio-tower tensors — gemma-4 e4b today. Auto-detected, no flags (mirrors vision). A request WITH audio on any other model gets an explicit 400 naming the model — never a silent text-only answer.
• Formats: WAV is decoded natively (PCM 16/24/32-bit + float32, any channel count / sample rate). mp3, m4a/AAC, FLAC, ogg(-CAF), AIFF and anything else CoreAudio reads transcode via macOS afconvert (src/audio/transcode.ts); undecodable bytes are a 400.
• Internal format: 16 kHz mono — multi-channel input is mean-mixed, other rates resample. Clips truncate at 30 s (the oracle processor’s 480 000-sample cap); one clip costs ceil(duration_ms / 40) prompt tokens, at most 750 (src/audio/decode.ts).
• Multiple clips per message and mixed image+audio requests work; media splices in document order.
• Routing: audio requests always run on the serial lane (never batched, even under --batch N) and skip the prompt cache — soft tokens are identical placeholder ids, so prefix matching across different clips would false-hit.
• The Anthropic surface (/v1/messages) has no audio block type in its protocol; audio blocks there are a 400 pointing back to this endpoint.

Non-streaming response:

{
  "id": "chatcmpl-…", "object": "chat.completion", "created": 1760000000,
  "model": "<loaded model id>",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "…",                      // null when the reply is ONLY tool calls
      "reasoning": "…",                    // present only when the model emitted a
                                           //   <think> channel (thinking-capable
                                           //   templates with thinking on)
      "tool_calls": [{                     // present when the model called tools
        "id": "call_…", "type": "function",
        "function": { "name": "…", "arguments": "{…json…}" }
      }]
    },
    "logprobs": { /* see below; only when requested */ },
    "finish_reason": "stop" | "length" | "tool_calls"
  }],
  "usage": {
    "prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0,
    "prompt_tokens_details": { "cached_tokens": 0 },  // prompt-cache reuse
    "lane": "serial",              // "serial" | "serial+spec" | "batched" —
                                    //   which execution lane served THIS
                                    //   request. Chat completions and text
                                    //   completions (streaming: final usage
                                    //   chunk).
    "speculation": {               // only when a draft source is mounted
      "drafted": 0,                //   draft tokens proposed
      "accepted": 0,               //   drafts accepted by the verify step
      "targetCalls": 0             //   target-model forward calls
    },
    "fill": {                      // only under MLX_BUN_FILL=strict, when a
                                    //   fill table was armed for the request
      "events": 0,                 //   injected spans
      "injected": 0,               //   tokens appended by the engine
      "strict": 0,                 //   …of those, from schema/template rows
      "echo": 0,                   //   …from the echo index (MLX_BUN_FILL=echo)
      "spanLens": [],              //   length of each span, in order
      "wastedSamples": 0,          //   in-flight samples dropped at an ASSERT fill
      "parseFallback": 0,          //   tool-call parses rejected after a fill
      "indexTruncated": 0,         //   spans cut by EOS / delimiter / budget / span
      "decodeSteps": 0,            //   tokens the model actually sampled
      "verifyEvents": 0,           //   verify-policy spans that reached the forward
      "verifyAccepted": 0,         //   …tokens the model's own argmax agreed with
      "verifyRejected": 0,         //   …tokens dropped and rewound
      "verifyUnsupported": 0,      //   verify spans dropped: caches cannot rewind
      "checkpointMs": 0,           //   time in cache checkpoint / rollback
      "branchStops": 0             //   echo spans stopped by a history fork
    }
  }
}

usage.lane is server-driven, never client-inferred (the web chat’s perf-strip lane badge reads it, not a heuristic): "batched" when --batch N picked up the request, "serial+spec" when a mounted draft source actually contributed accepted draft tokens to this reply, "serial" otherwise. It rides the SAME usage block on both streaming (final chunk before [DONE]) and non-streaming responses.

usage.completion_tokens counts the stopping EOS token in both native and speculative serial generation, matching mlx-lm. EOS is excluded from response content. An early callback stop or token budget does not count a later EOS that happened to be evaluated in the same speculative burst. An observed EOS reports finish_reason: "stop", including when its counted position equals max_tokens.

Speculative decoding is a server-level mode (serve --draft-model, or the model-free serve --draft-kind ngram, which mounts no draft model); there is no per-request draft field. The speculation usage extension appears on chat and text completions alike, non-streaming and on the final stream chunk. Spec-eligible requests are text-only on base weights. In the shared executor, Qwen grouped MTP, prompt lookup and standalone drafting support logprobs and bf16, uniform KV4/KV8 or per-layer affine KV, or TurboQuant. Qwen MTP and prompt lookup also support positive library conversion thresholds for uniform KV4/KV8. Per-layer mixed KV and adapters remain incompatible with speculation. Strict legacy serial execution retains its narrower bf16/KV4 policy. The KV4 compatibility control is MLX_BUN_QWEN_SPEC_KV4=0. Ineligible requests decode normally and omit the field. Qwen MTP reuses paired target/draft prefill and completed decode state through the shared RAM/SSD cache by default in v0.4.0, reported in cached_tokens. Configured SSD persistence supports reuse after restart; MLX_BUN_MTP_PROMPT_CACHE=0 disables reuse. Generated checkpoints cover processed tokens and use queued SSD persistence. Long-conversation performance acceptance remains open. Qwen prompt lookup also retains prefill and completed-output state in the common RAM/SSD cache, with its committed token history stored as a companion attachment. Standalone drafting on Qwen targets stores the draft model’s attention/recurrent state and pending last draft token as companion state, including restoration after restart. Qualified Qwen MTP, prompt lookup and standalone drafting use shared execution at one or several active rows and report usage.lane: "batched". Supported full-attention and rotating targets, including Llama, MiniCPM and Gemma, use shared bf16, uniform KV4/KV8 or per-layer affine KV or TurboQuant lookup and standalone drafting, with logprobs and generated RAM/SSD checkpoints. Delayed and per-layer affine rotating speculation uses the same row lifecycle; positive conversion thresholds are a library setting, while served schemes default to zero. Gemma assistant drafting also shares execution with bf16, uniform KV4/KV8 or per-layer affine KV or TurboQuant target storage and stores its last true target hidden alongside generated RAM/SSD prefixes. Donor attention handles row validity independently of the draft graph. DeepSpec and DSpark also use these shared target layouts and the common RAM/SSD cache for projected context, with independent confidence-based proposal lengths. Other provider/target combinations still use the serial executor. Prompt-lookup concurrent performance acceptance remains open. Ineligible requests can use continuous ordinary execution when their features support it, even while a draft is mounted.

usage.fill reports token fast-forwarding (MLX_BUN_FILL=strict|echo). Strict mode appends structural spans from the request’s tools and chat template at qualified parser boundaries, with no draft, verification or rollback. The model advances its hidden states and caches through the span, using its qualified append chunk size, then resumes sampling (design: speculative-decoding.md §“Token fast-forwarding”). Injected tokens are billed in completion_tokens like any other generated token; decodeSteps counts ordinary sampling steps, so injected / (injected + decodeSteps) is the share emitted through fill. Strict spans bypass sampling; echo spans still undergo sampled verification. Injected positions still run through the model layers. The field is absent when the feature is off or the request’s shape excludes it, including logprobs, structured output or media. Strict shared fill supports bf16, affine KV and TurboQuant layouts and retains absolute sampling positions for explicit seeds. Its usage.fill statistics use the same fields as explicit serial execution. Known tokens still advance model state. All-known steps skip intermediate vocabulary heads and use the model-qualified append geometry, including the specialized Qwen B1 append. Wider Qwen rows retain one-position arithmetic. Compiled replay, paging and interruption checkpoints remain outside this shared method. Explicit serial fill still excludes user-fixed seeds.

Under MLX_BUN_FILL=echo a second, weaker source joins: spans copied from earlier in the same session, carried under policy verify. Those ride the same single forward, are checked by the request’s sampler on its logits, and have their rejected tail rewound — so verifyAccepted + verifyRejected is what the echo index proposed and injected is what survived. wastedSamples counts only ASSERT fills (a verify fill consumes the in-flight sample as its first position’s check instead of discarding it).

Qwen MTP can alternate learned draft rounds with verified echo rounds. The provider consumes accepted copied tokens and target context before drafting again. Its asserted template spans remain ordinary sampled output. Learned draft counts, acceptance histograms and speculation.rounds exclude echo; targetCalls, tokensPerForward and forwardsSaved cover both round types. Echo activity remains in usage.fill, including matching prefixes established by ordinary MTP samples, so it is not additive with learned-draft counts. A request fully served by echoes can have zero learned drafts. Other mounted providers do not yet support this combination.

An explicit seed controls a request-local random stream and can use continuous execution. Repeating a request reproduces sampling at the same model arithmetic and batch composition. Different batch shapes can change floating-point results even with identical random draws; --batch 1 retains the existing fixed-shape control. This differs from mlx-lm’s policy of routing explicit seeds to its single-request generator.

logprobs / top_logprobs

With "logprobs": true and/or "top_logprobs": k (1–11), the non-streaming response’s choices[0].logprobs carries mlx_lm.server’s block, not OpenAI’s (server.py generate_response): entries are keyed by token id, token strings are the raw vocab pieces (convert_ids_to_tokens — e.g. "▁Hello"), and there is no bytes field. The same block appears on /v1/completions (mlx-lm shares the response builder; there is no legacy {tokens, token_logprobs} text-completion shape).

// top_logprobs: k > 0 (wins over logprobs when both are set — mlx-lm's if/elif)
"logprobs": { "content": [
  { "id": 9906, "token": "▁Hello", "logprob": -0.02,   // = top-1, merged from
    "top_logprobs": [                                   //   dict(top[0], top_logprobs=top)
      { "id": 9906, "token": "▁Hello", "logprob": -0.02 },
      { "id": 13347, "token": "▁Hi", "logprob": -4.1 }   // … k entries, sorted desc
    ] },
  // … one entry per generated token (reasoning/tool tokens included)
] }

// logprobs: true only
"logprobs": { "content": [ { "id": 9906, "logprob": -0.02 }, … ] }

The distribution matches mlx-lm generate_step exactly: full-vocab log-softmax of the logits after logits processors (logit_bias, penalties), before the sampler’s temperature/top-p/top-k/min-p/XTC. mlx-lm’s top-k order is unspecified (argpartition); ours is sorted descending — the same set, so the entry is deterministically the argmax. Stream chunks never carry logprobs (mlx-lm’s streaming responses don’t either). Requests with logprobs can use ordinary continuous execution under --batch N; other feature exclusions still apply. Invalid values are rejected with mlx-lm’s exact messages (see Errors).

Streaming

"stream": true is SSE: data: <chunk>\n\n per event, terminated by data: [DONE]. Chunks are chat.completion.chunk objects whose choices[0].delta carries {role: "assistant", content: ""} first, then {reasoning} increments while the model is inside its <think> channel, then {content} increments (multi-byte sequences are held back until decodable; text that could begin a stop sequence is held back until disambiguated, so no part of a stop sequence is ever streamed), then for tool calls a {tool_calls: [{index, id, type, function}]} delta; the last chunk carries finish_reason and usage. A generation error mid-stream is sent as data: {"error":{"message":…}} and the stream closes. While a large tool-call body is buffered for parsing, the server sends SSE comment heartbeats (: keep-alive) so client idle timers do not terminate the call.

Tool calls

Qwen’s <parameter=...> values are raw text. The parser preserves HTML entities, indentation and JSON string escapes, removing only one framing newline at either end. Array/object parameters are JSON-decoded according to the tool schema. The outer Qwen markup is parsed before generic JSON repair, so a nested edit array is not mistaken for the tool-call envelope.

Tool round-trip: send the assistant message with its tool_calls back, followed by { "role": "tool", "tool_call_id": …, "content": … } messages; multi-turn prompt prefixes reuse the KV prompt cache automatically.

Tool-call parsing is per model family (selectToolStreamMode in src/server.ts, parsers in src/tool-call.ts). Gemma 4 uses its native <|tool_call>…<tool_call|> sentinel tokens. Every other family parses decoded text once tools are active: OpenAI-JSON <tool_call>{…}</tool_call>, Qwen-style <tool_call><function=name><parameter=key>value</parameter></function></tool_call>, GLM-5.2’s name<arg_key>…<arg_value>…, and MiniCPM5’s <function name="…"><param name="…">…</param></function> (CDATA-wrapped values supported). Content before the tool markup still streams live; only the markup is withheld and converted to tool_calls. Argument values are decoded against the tool’s JSON schema (string-typed params stay strings).

When strict parsing fails, a repair pass runs before giving up: a ```json fence or prose prefix is stripped, single quotes and trailing commas are fixed, truncated braces are closed, a function name that landed inside arguments is lifted into the envelope, and Python literals True/False/None outside string values are coerced to true/false/null (pythonLiteralsToJson, src/tool-call.ts — Qwen3.5 in particular emits Python booleans). The repair is not reported on the wire; markup that still fails to parse falls back to plain content.

Structured output (grammar-constrained decoding)

Both /v1/chat/completions and /v1/completions accept the OpenAI/oMLX/vLLM structured-output fields (snake_case on the wire; the field names are GrammarRequest in src/grammar.ts):

{
  "response_format":              // OpenAI: {type:"json_object"} = any valid
    { "type": "json_schema",      //   JSON; {type:"json_schema"} constrains
      "json_schema": {            //   to the schema; {type:"text"} = no-op
        "name": "…", "schema": { /* JSON schema */ }, "strict": true,
        "any_whitespace": true } },  // false = compact separators (no
                                  //   whitespace freedom in the grammar —
                                  //   escape hatch for base models that
                                  //   greedily emit whitespace to max_tokens)
  "guided_grammar": "root ::= …", // raw EBNF grammar string (vLLM/oMLX)
  "guided_regex": "[A-Z][a-z]*",  // regex — the regex∩EBNF subset ONLY
                                  //   (classes, |, *, +, ?, parens);
                                  //   regex-only syntax (\d, ., anchors)
                                  //   hits the degrade path — a known gap
  "guided_choice": ["yes", "no"], // output is exactly one of these strings
  "structured_outputs": { /* … */ } // bare JSON schema (oMLX/vLLM alias)
}

Field	Compiles to (xgrammar)	Output
response_format: {type:"json_object"}	builtin JSON grammar	any valid JSON
response_format: {type:"json_schema", json_schema:{name, schema, strict?, any_whitespace?}}	compileJSONSchema(schema)	schema-conformant JSON
guided_grammar	Grammar.fromEBNF	text matching the grammar
guided_regex	EBNF-wrapped regex	text matching the regex subset
guided_choice	root ::= ("a" | "b" | …)	exactly one of the strings
structured_outputs	bare schema (alias for json_schema.schema)	schema-conformant JSON

Precedence when several are set (resolveGrammarRequest): guided_grammar

response_format json_schema > json_object > structured_outputs > guided_regex > guided_choice. response_format: {type:"text"} or no field at all means no constraint.

Enforcement is a per-step token bitmask at the sampler (xgrammar): invalid next tokens are masked to −inf after the logits processors, so valid-token numerics are untouched. Generation halts as soon as the grammar is satisfied (e.g. the closing } of a complete JSON object) with finish_reason: "stop". Works on both lanes — serial and --batch N (per-row matchers).

Degrade path — a grammar that fails to compile is never rejected (no 400/500): chat prepends a string-valued system instruction for the requested JSON schema, raw grammar, regex, or choice constraint, and the response carries a Warning header (grammar not enforced: …); /v1/completions has no chat template to inject into, so it emits the Warning header only. MLX_BUN_GRAMMAR=0 disables mask compilation but does not silently ignore a requested constraint: it uses this same degrade path with the hint grammar compilation disabled by MLX_BUN_GRAMMAR=0. Design and fidelity notes: docs/reference/server-api.md.

Errors

Memory estimates do not reject or shorten requests by default. Actual runtime failures use the request/stream error path when recoverable; native Metal allocation failure can terminate the process. Saved generation checkpoints remain available for an identical retry.

All errors are { "error": { "message": …, ... } }.

• 400 — malformed JSON (invalid JSON body), empty messages (messages required), unknown adapter id, vision request on a model without a sidecar (model has no vision sidecar), audio request on a model without an audio tower (message names the model), video on a non-Qwen3.5 model or combined with audio, prompt build failures (prompt build failed: …, which includes media fetch/transcode errors), non-numeric logit_bias keys/values (logit_bias must be a dict of int to float, mlx-lm’s coercion error), invalid logprobs params (mlx-lm’s exact validation: logprobs must be of type bool, top_logprobs must be of type int / at least 0 / at most 11; top_logprobs: -1 is the accepted “unset” sentinel), invalid reasoning_effort (reasoning_effort must be one of 'none', 'minimal', 'low', 'medium', 'high', 'xhigh').
• 400 with "type": "memory_admission", "code": "context_over_budget" — the PROMPT itself leaves no generation slot within an explicitly configured context/budget limit (src/serve/request-plan.ts). A prompt that fits is never rejected for a broad max_tokens: the upper bound is capped to the remaining room (max_tokens is a ceiling, not a promise) and generation proceeds. This is opt-in via an operator budget/context cap (or the fixed GLM-5.2 layout), never enforced from default fit estimates. /stats.admission.enforced_context_tokens is null when no cap applies.
• 429 with "type": "resource_admission", "code": "queue_full" when a preparation or generation queue has 64 waiting requests. If SSE headers are already sent, the stream emits its terminal error instead.
• 500 — a generation failure after admission; the stack is logged server-side, the body keeps only the message.

POST /v1/completions (raw text completion)

mlx_lm.server’s text-completion endpoint: no chat template — the prompt string is tokenized directly (the tokenizer’s own BOS handling, exactly mlx-lm’s tokenizer.encode(request.prompt)) and the model continues it. Same generation gateway, admission control, prompt cache, and adapter selection as chat.

{
  "prompt": "Once upon a time",  // REQUIRED, non-empty string only (token
                                 // arrays rejected, matching mlx_lm.server)
  "max_tokens": 512,             // unset = no cap: generate until EOS or the
                                 // an explicit operator limit (DEVIATION: mlx_lm.server
                                 // defaults 512 — pass --max-tokens 512 to
                                 // reproduce); max_completion_tokens wins
  "stream": false,
  "stop": "\n\n",                // string or array, decoded-text matching
  // plus every sampling/penalty field from /v1/chat/completions:
  // temperature, top_p, top_k, seed, min_p, xtc_probability,
  // xtc_threshold, logit_bias, repetition_penalty,
  // repetition_context_size, presence_penalty, presence_context_size,
  // frequency_penalty, frequency_context_size, hlg, adapter,
  // logprobs (BOOL, mlx-lm's type — not OpenAI's legacy int),
  // top_logprobs (0-11 or -1) — same response block as chat, see
  // "logprobs / top_logprobs" above (mlx-lm shares the builder),
  // response_format / guided_grammar / guided_regex / guided_choice /
  // structured_outputs — structured output, same semantics and
  // precedence as chat (see "Structured output" above); the degrade
  // path here emits the Warning header only (no template to inject a
  // system prompt into)
}

Response is the OpenAI text-completion object:

{
  "id": "cmpl-…", "object": "text_completion", "created": 1760000000,
  "model": "<loaded model id>",
  "choices": [{ "index": 0, "text": "…", "finish_reason": "stop" | "length",
                "logprobs": { /* only when requested */ } }],
  "usage": { "prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0,
             "prompt_tokens_details": { "cached_tokens": 0 },
             "lane": "serial" }
             // + "speculation" under --draft-model, as on chat
}

Streaming ("stream": true) is SSE: text_completion chunks whose choices[0].text carries the delta; the final chunk carries finish_reason and usage (always attached — mlx-lm gates usage behind stream_options.include_usage; ours is an additive superset), then data: [DONE]. There is no echo parameter (mlx_lm.server has none). 400 on a missing/empty/non-string prompt, plus the same logit_bias, logprobs, and admission errors as chat.

POST /v1/messages (Anthropic Messages API)

Anthropic-protocol surface over the same engine — on by default, like optiq serve. Point any Anthropic-SDK tool at the server (ANTHROPIC_BASE_URL=http://localhost:8080, any x-api-key) — Claude Code works as a client this way. Translation lives in src/anthropic.ts; the request funnels into the chat handler above.

• system (string or text blocks), messages with string or content-block arrays; tool_use / tool_result blocks map to the native tool-calling path; image blocks (base64 or url source) hit the vision path on sidecar models, including image blocks nested inside a tool_result (the computer-use shape). Other image source types are a 400. The protocol has no audio input block; audio-shaped blocks (input_audio / audio / audio_url) are rejected with a 400 pointing to /v1/chat/completions. Prior-turn thinking blocks are dropped on re-ingest.
• tools ({name, description, input_schema}) map to function tools; entries without name + input_schema (server-tool types such as web_search) are dropped silently.
• max_tokens, temperature, top_p, top_k, stop_sequences, stream as in the Anthropic spec. The mlx-lm sampler/penalty extensions (min_p, xtc_probability, xtc_threshold, repetition_penalty, presence_penalty, frequency_penalty + their *_context_size windows) pass through as extras; logit_bias does not exist in this protocol and is not accepted here.
• Response: {id: "msg_…", type: "message", role: "assistant", content: [{type: "text"} | {type: "tool_use"}…], stop_reason, stop_sequence, usage: {input_tokens, output_tokens, cache_read_input_tokens?}} — cache_read_input_tokens is present when the prompt cache contributed.
• Streaming follows the Anthropic event grammar exactly: message_start → content_block_start/delta/stop (text_delta, input_json_delta) → message_delta (stop_reason + usage) → message_stop. Errors are event: error frames.
• Errors: {type: "error", error: {type: "invalid_request_error" | "api_error", message}} — the chat handler’s 4xx/5xx are re-wrapped into this shape.

POST /v1/responses (OpenAI Responses API)

Responses-protocol surface (Codex, Cursor, Continue, Cline, and the OpenAI SDK speak this now). Oracle: optiq responses shim; port in src/responses.ts.

• input (string or item array: message, function_call, function_call_output), instructions (merged with any system/developer items into one leading system message), max_output_tokens (or max_tokens), temperature, top_p, top_k, flat tools/tool_choice (built-in tool types — web_search, file_search, mcp, computer_use — dropped silently; {type:"function", name} maps to the chat object form, "auto"/"none"/"required" pass through), stream. The mlx-lm sampler/penalty extensions (min_p, xtc_probability, xtc_threshold, logit_bias, repetition_penalty, presence_penalty, frequency_penalty + *_context_size) pass through as extras.
• previous_response_id resumption: pass a prior response id instead of resending the conversation; the server splices the stored input + output back in (instructions carry forward when omitted; reasoning items are not replayed). Store is per-process, 1 h TTL, 32 MiB byte-capped LRU — observable at GET /stats (response_store). Unknown/expired id → 404.
• Response: {id: "resp_…", object: "response", status: "completed" | "incomplete", output: [{type: "reasoning", summary: [{type: "summary_text", text}]}?, {type: "message"}, {type: "function_call"}…], usage: {input_tokens, output_tokens, total_tokens, output_tokens_details: {reasoning_tokens}}, previous_response_id}. A reasoning item is present only when the model emitted a think channel.
• Streaming event chain: response.created → response.in_progress → response.output_item.added → response.content_part.added → response.output_text.delta… → response.output_text.done → response.content_part.done → response.output_item.done → response.completed (+ response.function_call_arguments.delta/.done for tool calls; a reasoning output item precedes the message when the model thinks).
• Errors: {error: {message, type: "invalid_request_error" | "server_error", param: null, code: null}}.

POST /v1/embeddings (OpenAI Embeddings API)

Text embeddings, available when the served model is an embedding model (plain Qwen3 / Qwen3-Embedding) — consistent with the single-model server: mlx-bun serve mlx-community/Qwen3-Embedding-4B-4bit-DWQ, then POST here. On any other served model the route returns 400 invalid_request_error (src/serve/model-admin-routes.ts).

• input (string or array of strings; anything else is a 400), model (ignored — the served model is used), and a non-standard instruction (optional): applies Qwen3-Embedding’s query format Instruct: {instruction}\nQuery:{text}, which steers which similarity axis the geometry reflects. Omit it for plain document embeddings.
• Pooling is last-token + L2-normalization (the vectors are unit-length); similarity is the dot product. Bit-exact vs the mlx-lm qwen3 reference (tests/parity/qwen3-embed-parity.test.ts).
• Response: { "object": "list", "data": [{ "object": "embedding", "index": 0, "embedding": [float…] }, …], "model": "<served id>", "usage": { "prompt_tokens": N, "total_tokens": N } }. Embedding is a single forward (no decode loop), so it runs inline — not through the generation gateway.

curl localhost:8080/v1/embeddings -H 'content-type: application/json' \
  -d '{"input": ["the cat sat on the mat", "a kitten by the window"]}'

POST /v1/audio/transcriptions · POST /v1/audio/translations (speech-to-text)

OpenAI-compatible Whisper transcription (translations = the same route with task translate, English output). Served by every mlx-bun serve beside the chat model — the Whisper checkpoint (--whisper-model, default: the first downloaded whisper model) loads for each take and is released right after it unless --whisper-idle-unload/--whisper-resident say otherwise — and by the transcription-only server that serve <whisper checkpoint> starts (server-config.md). Model support and the parity contract: models.md; CLI equivalent: mlx-bun transcribe.

Request — multipart/form-data (OpenAI’s shape) or JSON:

Field	Meaning
file	audio upload (multipart) or base64 / data: URL string (JSON; input_audio.data and audio are accepted aliases). WAV decodes through the exact PCM parser; any other CoreAudio-readable container (mp3/m4a/aac/flac/ogg/aiff/caf, mp4/mov audio tracks…) decodes in-process through AudioToolbox (src/audio/audiotoolbox.ts, 3.7 ms for an 11 s mp3). Undecodable bytes are a 400.
model	ignored — the configured Whisper checkpoint is used
language	ISO code or name; omitted / auto detects from the first 30 s
prompt	initial prompt (vocabulary, style); the last 223 tokens are kept
response_format	json (default) · verbose_json · text · srt · vtt
temperature	one temperature in [0, 1]; omitted = mlx-whisper’s (0, 0.2, …, 1.0) fallback ladder
stream	true → server-sent events (below)
timestamp_granularities[]	segment (default); word adds words (word, start, end, probability) to verbose_json segments via cross-attention DTW on the checkpoint’s alignment heads — one extra decoder pass per window
beam_size	non-standard: beam search width (default greedy); requires temperature 0
vocabulary	non-standard: array (JSON) or comma-separated (multipart) recognition hints, fitted whole-term into the 223-token prompt budget after prompt; usage is reported
condition_on_previous_text	non-standard, default true
no_speech_threshold, without_timestamps	non-standard mlx-whisper decode options
vad, vad_threshold, vad_min_speech_ms, vad_trim	non-standard: Silero VAD gate (v6.2, ggml-org/whisper-vad weights in the HF cache). No detected speech → {"text": ""} with mlx_bun.vad.speech = false and Whisper never runs (≈25 ms for 11 s). vad_trim crops to the speech span ± 0.5 s first; timestamps stay clip-relative. Segments are reported in mlx_bun.vad.segments (seconds). Defaults: threshold 0.5, min speech 250 ms
faithful	non-standard: true runs the oracle-parity graph instead of the fast path (fused attention, compiled decoder step); same tokens up to documented near-ties, slower
audio_ctx	non-standard, Lab: encode only the first n of 1500 encoder positions (whisper.cpp -ac). Cheaper for short clips; measured hallucinated repeats below ~1024 on the fixture clips — off unless set

Response — json: { "text": "…", "mlx_bun": { "model", "language", "duration", "timings": { "load_ms", "transcribe_ms", "total_ms" }, "vocabulary"?: { "included", "omitted", "token_count", "token_budget" } } }. verbose_json adds task, language, duration and openai-whisper’s segments (id, seek, start, end, text, tokens, temperature, avg_logprob, compression_ratio, no_speech_prob). text/srt/vtt return the bare body (text/plain, application/x-subrip, text/vtt) with x-mlx-bun-language and x-mlx-bun-timings headers. load_ms is non-zero exactly when the request paged the weights in.

Streaming (stream=true): text/event-stream with transcript.text.delta { "delta": "<segment text>", "segment": { "id", "start", "end" } } per finalized segment, transcript.progress { "done", "total" } (mel frames), and a final transcript.text.done carrying the non-streaming body.

Errors: 400 invalid_request_error (fields, undecodable audio, clips shorter than 0.1 s), 415 (content type), 499 on client cancel, and 503 model_unavailable when no Whisper checkpoint is on disk (the message carries the mlx-bun get command).

POST /admin/transcription/unload pages the weights out now and returns { "unloaded": bool, "resident", "loads", "unloads", "requests", "last_load_ms", "idle_unload_sec" } — the same block /health and /stats report as transcription on the transcription-only server. Transcriptions run one at a time and never overlap chat generation on the GPU (the generation gateway’s exclusive lock).

curl localhost:8080/v1/audio/transcriptions -F file=@meeting.m4a -F language=en -F response_format=srt
curl localhost:8080/v1/audio/transcriptions -H 'content-type: application/json' \
  -d '{"file":"<base64 wav>","beam_size":5,"vocabulary":["Sotto","SwiftUI","Metal"]}'

Streaming transcription sessions (/v1/audio/sessions)

For audio that is being recorded while it is sent (dictation). Every completed 30 s window is transcribed as soon as it exists, under the same exclusive lock as everything else, so finish costs one window regardless of take length (measured: a 62 s take that needs 1165 ms in batch returns 327 ms after its last chunk, identical text — M1 Max). With vad: true the Silero gate runs on the stream: nothing is transcribed until speech has been seen, and a take with no speech finishes as {"text": ""} without ever running Whisper. The one-shot parameters (language, prompt, vocabulary, beam_size, temperature, vad*, timestamp_granularities, faithful) apply; response_format is json or verbose_json.

Route	Body / response
POST /v1/audio/sessions	JSON params → { "id", "model", "vocabulary" } (the weights load here if they were paged out)
POST /v1/audio/sessions/<id>/audio	one chunk: content-type: audio/pcm;rate=16000 raw little-endian float32 mono (sotto’s format, no decode), or any CoreAudio container. → { "samples", "duration", "speech", "segments": [...] } with every segment finalized so far
POST /v1/audio/sessions/<id>/finish	→ the one-shot response body (json/verbose_json with mlx_bun), closes the session
DELETE /v1/audio/sessions/<id>	discard, 204

Chunk order is the caller’s responsibility (send sequentially and await each receipt). Unknown ids are 404; a finished session is 409; at most 64 sessions are open at once (429). Windows fed early are normalized against the audio received so far (the log-mel clamp is relative to the clip maximum), which can differ from a batch transcription in near-silent mel bins — the fixture takes decode identically.

GET /v1

API index for discovery tooling:

{ "name": "mlx-bun", "version": "<package version>", "model": "<served model id>",
  "endpoints": ["POST /v1/chat/completions", "POST /v1/completions", "POST /v1/messages",
                "POST /v1/responses", "POST /v1/embeddings", "POST /v1/audio/transcriptions",
                "POST /v1/audio/translations", "GET /v1/models",
                "GET/POST/DELETE /v1/adapters", "GET /health", "GET /stats", "GET /fit",
                "GET /library", "GET /downloads"] }

(The list is a curated subset; this document is the full inventory.)

GET /v1/models

{ "object": "list", "data": [{ "id": "<model id>", "object": "model", "created": <unix s>, … }] } — created is the server start time for every row.

The served model is FIRST, with extra capability fields:

{
  "id": "<served model id>", "object": "model", "created": 1760000000,
  "owned_by": "mlx-bun",
  "context_window": 32768,
  "reasoning": false,           // switchable thinking channel (gates the
                                 //   web chat's thinking toggle)
  "vision": false,               // image input accepted (tower loaded or
                                 //   lazily loadable)
  "audio": false,                // input_audio content parts accepted
                                 //   (tower loaded or lazily loadable)
  "batch_mode": "batch",       // loaded cache capability; per-request
                                 //   usage.lane is authoritative
  "tools": true,
  "structured_output": true,
  "embeddings": false,
  "adapters": false,
  "training": false,
  "dsa": true,                  // GLM-5.2 only; false elsewhere
  "mtp": true,                  // checkpoint-native GLM MTP is mounted
  "capabilities": {             // explicit discoverable surface
    "chat_completions": true, "text_completions": true,
    "anthropic_messages": true, "responses": true,
    "streaming": true, "tools": true, "structured_output": true,
    "logprobs": true, "embeddings": false,
    "vision": false, "audio": false,
    "adapters": false, "training": false
  },
  "gen_defaults": {              // model-author sampling defaults
    "temperature": 0.7,          //   (generation_config.json, with any
    "top_p": 0.95,               //   --temperature/--top-p/--top-k server
    "top_k": 0                   //   override applied); null when neither
                                 //   the model nor the server set one
  }
}

Followed by every other servable model the local registry knows (mlx-lm scans the HF cache here; the registry is that scan, filtered to supported architectures, one row per repo via listCanonical() — duplicate snapshots from upstream re-pushes are not separate entries):

{ "id": "<repo id>", "object": "model", "created": 1760000000,
  "vision": false,               // visionCapable(): sidecar OR encoder-free
                                 //   unified vision config
  "tier": "targeted" | "generic" // supportTier(): dedicated/generated forward
                                 //   ("targeted") vs the Tier-0 universal
                                 //   module ("generic"); entries with no tier
                                 //   (unsupported) are filtered out entirely
}

GET /v1/models/<id> filters the list to that id — same list shape, matching mlx_lm.server.

vision / audio say whether the served model accepts image_url / input_audio content parts (tower loaded or lazily loadable — see Audio input above for what audio requires), so clients can discover capabilities instead of probing for a 400. The served-model row also includes batch_mode: "off" | "serial" | "batch": the configured cap and the loaded model’s cache capability together. Per-turn usage.lane remains authoritative for a particular request; for example, native-MTP requests route serial even when the served GLM model reports batch_mode: "batch" for ordinary requests. For GLM-5.2, the non-generative fields are deliberately false: embeddings, vision/audio, adapters, and training are not emulated by the serving port.

GET /health

{"status": "ok"} — byte-for-byte what mlx_lm.server returns.

GET /stats

{
  "server": { "owner": "serve" | "pi-session" | "embedded", "model": "...", "started_at": 0 },
  "prompt_cache": { "entries": 0, "bytes": 0, "max_bytes": 0, "hits": 0, "misses": 0, "object_hits": 0, "object_misses": 0, "object_restores": 0 },
  "response_store": { "entries": 0, "bytes": 0, "max_bytes": 33554432, "ttl_ms": 3600000 },
  "kv_quant": { "mode": "bf16" | "uniform-kv8" | "turbo k8v3" | "mixed (kv_config.json)",
                 "layers": { "kv4": 8, "bf16": 40 },     // turbo: { "turbo-k8v3": 8, "bf16": 40 }
                 "attention": { "global": 10, "sliding_window": 38 }, "recurrent_layers": 0 },
  // present only when --ssd-cache is on:
  "ssd_cache": { "dir": "...", "entries": 0, "bytes": 0, "max_bytes": 0,
                 "restores": 0, "spills": 0, "restore_ms_last": 0, "demotions": 0,
                 // bounded write-behind queue (MLX_BUN_SSD_SPILL_QUEUE_GB):
                 "pending_snapshots": 0, "pending_spills": 0,
                 "pending_spill_bytes": 0, "dropped_spills": 0,
                 "failed_spills": 0, "longest_durable_prefix_tokens": 0 },
  "admission": {
    "max_safe_context": 0,            // advisory fit estimate, tokens
    "enforced_context_tokens": null,  // explicit cap or fixed layout; null = none
    "memory_budget_bytes": null,      // explicit budget, or null
    "usable_bytes": 0,
    "weights_bytes": 0
  },
  // present only for the direct Colibri GLM-5.2 runtime:
  "glm52": {
    "preset": "g5-32gb-quality",
    "planned_process_bytes": 0,
    "process_limit_bytes": 0,
    "context_tokens": 4096,
    "max_generation_tokens": 128,
    "batch_size": 1,
    "dsa": true,
    "mtp": true,
    "mtp_draft_tokens": 3,
    "resident_weight_bytes": 0,
    "main_expert_slab_bytes": 0,
    "mtp_expert_slab_bytes": 0,
    "expert_runtime": {               // null when the expert runtime is off
      "main_residency": { "resident": 0, "pinned": 0, "working": 0,
                            "hits": 0, "misses": 0, "evictions": 0 },
      "mtp_residency": { /* same shape */ } | null,
      "last_turn": { "main": { "demand": { "hits": 0, "misses": 0, "readBytes": 0 } },
                     "mtp": { "demand": { "hits": 0, "misses": 0, "readBytes": 0 } },
                     "pilot": null },
      "last_repin": []
    }
  },
  "batch": {
    "configured": 1,                  // the --batch N value
    "mode": "off",                   // "off" | "serial" | "batch": configured
                                           // and model-cache capability together
    "batched": false,                 // true only when mode == "batch"
    "active_rows": 0,                 // rows currently decoding in the batch
    "pending_rows": 0,                // queued + mid-prefill rows waiting
    "submitted_rows": 0,              // cumulative rows routed to the batch lane since start
    "kv_bytes": 0,                    // projected aggregate KV bytes of admitted rows
    "kv_budget_bytes": null           // --kv-budget cap, or null (uncapped)
  }
}

kv_quant.mode reflects the --kv-quant flag the server was started with; the default is bf16 (no KV quantization). mixed (kv_config.json) appears only when --kv-quant config was passed explicitly and the checkpoint ships a per-layer kv_config.json — per-layer KV quantization is never selected automatically.

kv_quant.layers counts configured attention KV codecs; attention separates full and sliding attention. recurrent_layers counts linear/recurrent layers whose convolution and recurrent state retain their own precision. They do not count as bf16 or quantized attention KV. These are model/policy counts, not live allocation measurements or a per-request conversion-progress report.

pending_snapshots counts prompt-cache entries scheduled for persistence but not yet confirmed by the atomic SSD store. longest_durable_prefix_tokens is the longest prefix already present in the SSD index, not merely queued.

GET /fit

Fit assessment for the loaded model on this machine, plus a capability matrix across Apple SKUs. Used by the status page. Experts bytes come from the registry so MoE active-parameter predictions match the serve banner and mlx-bun fit. When the eval DB has a real measured decode rate for this model snapshot, it is included and takes precedence over the prediction.

For the direct Colibri GLM-5.2 runtime this endpoint switches to the exact serve-time memory plan. The response separates the full streamed artifact on disk from resident weights, main/MTP expert slabs, KV, and other reserves. measured_decode_tps is the G5 warm quality-preserving measurement; glm52.direct_oracle_warm_decode_tps is the same-machine direct Colibri control, while glm52.aspirational_decode_tps is a target, not a release gate. The single-row SKU result describes the current machine because the generic all-resident SKU estimator is inapplicable (report.predicted_decode_tps is null there).

{
  "machine": { "chip": "M4 Pro", "ram_bytes": 0, "bandwidth_gbs": 0.0 },
  "context_tokens": 8192,          // estimated context capacity
  "typical_context_tokens": 8192,  // min(8192, context_tokens)
  "typical_decode_tps": 0.0,       // predicted at typical_context_tokens
  "measured_decode_tps": null,     // real number from eval DB, or null
  "measured_at": null,             // unix ms of measurement, or null
  "report": {
    "fits": true,
    "weights_bytes": 0,
    "kv_bytes": 0,
    "transient_bytes": 0,
    "total_bytes": 0,
    "usable_bytes": 0,
    "max_safe_context": 8192,
    "predicted_decode_tps": 0.0
  },
  // GLM-5.2 only:
  "glm52": {
    "artifact_disk_bytes": 0,       // exact registry size; null if unavailable
    "main_expert_slab_bytes": 0,
    "mtp_expert_slab_bytes": 0,
    "max_generation_tokens": 128,
    "direct_oracle_warm_decode_tps": 0.27,
    "aspirational_decode_tps": 2.0
  },
  "sku_matrix_ctx": 32768,
  "sku_matrix": [{
    "sku": "M4 Pro 24 GB", "ram_gb": 24,
    "fits": true, "max_context": 32768, "decode_tps": 0.0
  }]
}

Adapters (LoRA hot-swap)

Routes in src/serve/model-admin-routes.ts; the manager is AdapterManager in src/lora.ts. Requests selecting the same ordered adapter set can share a continuous group. Different sets wait for the current group to finish and then use the same executor. Prefix-cache identity is resolved under the execution lease, so replacing an adapter while a request is queued cannot reuse state from its previous weights.

• GET /v1/adapters — { adapters: [{ id, path, rank, scale, size_bytes, mounted_layers, ram_bytes }] } — currently-mounted adapters only. ram_bytes is the actual resident size of the adapter’s mounted lora_a/lora_b arrays (summed MlxArray.nbytes — real RAM cost while mounted, not a guess from the on-disk file size).
• GET /v1/adapters/available — { adapters: [{ id, path, rank, scale, base_model, mounted, compatible }] } — every adapter found on disk (~/.cache/mlx-bun-finetunes, ~/.cache/mlx-bun/adapters), unfiltered. compatible is true when the adapter’s recorded base model matches the currently-served model (compared by bare repo name) or the adapter recorded no base at all; mounted is true when it’s already loaded. The web chat’s adapter chip uses compatible to gray out entries it won’t let you select rather than hiding them.
• POST /v1/adapters — { "id": "...", "path": "/dir" } (both required, else 400 "id and path required"); mounts through the generation queue (never races a forward pass). Response: { id, mounted_layers, rank, scale, ram_bytes }. 400 on shape/compat mismatch — validation is all-or-nothing.
• DELETE /v1/adapters/<id> — unmount; { id, removed_layers } on success, 404 {"error":{"message":"adapter <id> not mounted"}} otherwise.

Select per request with the adapter body field (chat, text completions; the Anthropic and Responses surfaces pass it through as an extra), which also accepts a composed spec — "a+b" (or "a,b") stacks two mounted adapters, their LoRA residuals summed in order (AdapterManager.resolveSpec / parseAdapterSpec in src/lora.ts); every named id must already be mounted or the request 400s with the unknown id named. Prompt-cache entries are namespaced per adapter spec (including composed ones), so switching adapters — or switching which ones are stacked — never reuses another combination’s KV.

serve --adapter <dir> (alias --adapter-path, mlx_lm.server’s spelling) mounts an adapter at startup through this same machinery and makes it the default for requests that send no adapter field; an explicit adapter (including "none") always wins, and hot-swap via these endpoints is unchanged.

Cache session affinity

Completion requests may supply session_id or prompt_cache_key in the JSON body, or x-session-affinity / session_id in a header. Precedence is body session_id, body prompt_cache_key, x-session-affinity, then session_id. This applies to chat completions, text completions, Responses and Messages. The shared mlx-bun Pi provider sends session-affinity headers automatically.

A session indexes its latest published checkpoint within each numerical cache namespace. Compatible continuations use that checkpoint directly; edited, branched or missing histories use ordinary prefix matching. The full intended input is still required: a session label does not append omitted messages, isolate access, change model settings or bypass token compatibility.

The cache prefers session-referenced checkpoints when choosing RAM residents, but can evict them when required. SSD persistence remains independent. Session indexes are process-local; after restart, the first request uses ordinary SSD lookup and re-establishes the association. Entries remain shareable across sessions and requests without session metadata. MLX_BUN_SESSION_CACHE provides the A/B control described in server-config.md.

GET /stats adds prompt_cache.session_hits, session_misses and prefix_scans. A session hit counts direct RAM checkpoint selection, including state prepared from SSD before execution. A session miss counts execution lookups with session metadata that need ordinary candidate search. Prefix scans count ordinary execution searches, not tokenization or preparation probes.

Qwen image/video encoder reuse reports prompt_cache.object_hits, object_misses and object_restores. A hit borrows an exact RAM tensor object; a miss may restore from SSD or recompute. Object entries and bytes are included in the existing cache totals and share its RAM budget and persistence queue. These counters do not count language-model KV prefix hits. Media prompt KV reuse remains disabled because token IDs alone do not identify media contents.

POST /admin/cache/session/close

Send { "session_id": "agent-42" } to remove that session’s cache association and retention preference. Returns { "closed": true } when an ID is supplied, or { "closed": false } otherwise. Closing does not delete checkpoints from RAM or SSD. Applications can close idle or completed sessions; a later request with the same ID establishes a new association.

POST /admin/cache/flush

Forces every pending SSD prompt-cache snapshot through the serial write queue and waits for the atomic temp-file/fsync/rename boundary. The response is 200 only when no snapshot or spill remains pending and no write failed during this flush; otherwise it is 503 with the counters that explain why. A snapshot missing from both RAM and usable SSD coverage remains pending across repeated flushes. Retrying does not turn a missing snapshot into a successful durability result. Already-durable prefixes avoid another clone and write.

{
  "durable": true,
  "flushedSnapshots": 1,
  "missingSnapshots": 0,
  "pendingSnapshots": 0,
  "pendingSpills": 0,
  "pendingSpillBytes": 0,
  "droppedSpills": 0,
  "failedSpills": 0,
  "elapsedMs": 842,
  "entries": 2,
  "longest_durable_prefix_tokens": 16384
}

The standard benchmark calls this endpoint before its restart leg and records the result. Normal mlx-bun serve shutdown on SIGINT or SIGTERM performs the same flush after active requests drain, with a 120-second default timeout. Without --ssd-cache the route still answers (entries: 0).

GET /library

?refresh=1 bypasses the short response cache, including after a parent-managed job finishes.

Returns all models found in the local HuggingFace hub cache (via the registry scan), each annotated with a fit assessment for this machine. Response is cached for 30 seconds (registry scan + config reads; no tensor bytes are read). Handler: src/serve/discovery-routes.ts.

{
  "models": [{
    "repo_id": "…",
    "model_type": "gemma3" | "minicpm5" | "qwen3" | …,
    "size_bytes": 0,
    "quant_bits": 4,
    "vision": false,             // visionCapable(): sidecar OR encoder-free
                                 //   unified vision config (not sidecar-only)
    "audio": false,              // audio-capable (audio_config + sidecar audio-tower tensors)
    "supported": true,           // supportTier() !== null (recognized family)
    "support_tier": "targeted" | "generic" | null,  // dedicated/generated
                                 //   forward vs the Tier-0 universal module;
                                 //   null only when supported is false
    "serving": false,            // currently loaded in this server
    "assessment": {              // null if config unreadable
      "fits": true,
      "max_safe_context": 8192,
      "predicted_decode_tps": 0.0
    }
  }]
}

One row per repo (listCanonical(); duplicate snapshots from upstream re-pushes surface only via the CLI’s mlx-bun ls --all-revisions).

GET /downloads

Snapshot of the last 5 model downloads (active, completed, or errored) initiated via mlx-bun download, the web library panel, or POST /api/hub/download.

{
  "downloads": [{
    "repo_id": "…",
    "state": "active" | "done" | "error",
    "current_file": "model.safetensors" | null,
    "received_bytes": 0,
    "total_bytes": 0,
    "files_done": 0,
    "files_total": 0,
    "bytes_per_sec": 0,          // rolling ~5 s window
    "started_at": 1760000000,
    "finished_at": null,
    "error": "…"                 // present on state "error"
  }]
}

Jobs (/api/jobs)

The web wizards (quantize, fine-tune, dataset) run as jobs: a SQLite row plus an NDJSON event log (src/jobs/). Submit routes return a job_id; these routes read progress. Loopback-served admin routes, no separate auth (same posture as every other /api/* route). Handler: src/serve/admin-routes.ts.

GET /api/jobs?limit=50&kind=quantize|finetune|dataset — most recent jobs, newest first: { "ok": true, "jobs": [JobRow, …] }.

GET /api/jobs/<id> — one row, or 404 { "ok": false, "error": "job not found" }:

{ "ok": true, "job": {
  "id": "…", "kind": "quantize" | "finetune" | "dataset",
  "status": "queued" | "running" | "done" | "failed" | "zombie",
  "config_json": "{…}",          // the submit body as stored
  "progress": 0.0,               // 0..1, from the latest stage/metric event
  "message": "…" | null,
  "log_path": "…", "output_path": "…" | null, "error": "…" | null,
  "started_at": "<ISO>", "ended_at": "<ISO>" | null
} }

GET /api/jobs/<id>/stream — EventSource-compatible SSE tail of the job’s log: a retry: 1500 line first, then one data: <JobEvent> frame per event, polling while the job is non-terminal, then event: end + data: {} and close. An unknown id streams a single failed event then end. Events:

{ "type": "started", "ts": 0 }
{ "type": "log", "line": "…" }
{ "type": "stage", "stage": "…", "progress": 0.5, "message": "…" }
{ "type": "metric", "kind": "train" | "val", "step": 0, "loss": 0.0 }   // + grad_norm, learning_rate, tokens_per_sec, …
{ "type": "done", "ts": 0, "output_dir": "…", "summary": { } }
{ "type": "failed", "error": "…", "ts": 0 }

Fine-tune (/api/finetune/*)

The HTTP entry point to LoRA training (src/serve/lab-routes.ts → src/train/job.ts). It runs the same runner the CLI’s mlx-bun train drives, as a subprocess job (crash isolation + one GPU lease across concurrent server requests). Data formats, every config knob, and the methodology are in training.md; this section is the wire shape.

POST /api/finetune/submit — body is FinetuneSubmit (src/train/job.ts). model_dir and data_dir are required at the route (400 "model_dir and data_dir required"); adapter_path defaults to ~/.cache/mlx-bun/adapters/adapter-<timestamp> when omitted. Everything else is optional and falls back to DEFAULT_TRAIN_CONFIG:

{
  "model_dir": "/path/to/snapshot",       // required
  "data_dir": "/path/to/dataset",         // required: train.jsonl (+ optional valid.jsonl)
  "adapter_path": "/path/to/output",      // optional (default above)
  "method": "sft" | "dpo" | "orpo",
  "rank": 16, "scale": 20, "rank_scaling": "…", "target_modules": ["…"], "num_layers": 0,
  "iters": 300, "learning_rate": 2e-4, "max_seq_length": 2048, "batch_size": 1,
  "grad_accumulation_steps": 1, "seed": 0, "steps_per_report": 1, "steps_per_eval": 0,
  "weight_decay": 0.0, "lora_dropout": 0.0, "rs_lora": false, "lora_plus_ratio": 1,
  "grad_checkpoint": false, "mlp_split": false, "segment_size": 0,
  "save_checkpoints": true, "grad_clip_norm": 0, "val_max_examples": 0,
  "dpo_beta": 0.1, "dpo_warmup_iters": 0, "dpo_lr_schedule": "constant" | "cosine",
  "orpo_lambda": 0.1, "orpo_warmup_iters": 0, "orpo_lr_schedule": "constant" | "cosine",
  "orpo_chunk_size": 0, "orpo_fused_ce": true, "orpo_flash_ce": true, "orpo_prefix_shared": true,
  "sft_scope": "full" | "response",
  "warm_start_adapter": "/path/to/adapter"
}

Response: { "ok": true, "job_id": "…", "adapter_path": "…" }. Then follow GET /api/jobs/<job_id>/stream for metric events (kind: "train"|"val", step, loss, …) and the terminal done/failed. Config validation errors (an invalid sft_scope, a missing adapter_path at the runner) surface as the job’s failed event, not as a 4xx.

curl -s localhost:8080/api/finetune/submit -X POST -H 'content-type: application/json' -d '{
  "model_dir": "/path/to/snapshot", "data_dir": "/path/to/dataset",
  "adapter_path": "/path/to/output-adapter",
  "method": "sft", "rank": 16, "iters": 300, "learning_rate": 2e-4, "max_seq_length": 2048
}'
# → {"ok":true,"job_id":"…","adapter_path":"/path/to/output-adapter"}
curl -N localhost:8080/api/jobs/<job_id>/stream

POST /api/finetune/inspect-dataset with {"path": "<data_dir>"} — probe a dataset directory before submitting: { "ok": true, "n_train": N, "n_valid": N, "format": "messages" | "prompt-completion" | "text" | "dpo" | "unknown" }, or { "ok": false, …, "error": "<train.jsonl> not found" } (HTTP 200 either way).

POST /api/finetune/merge with {"adapter_a": "/dir", "adapter_b": "/dir", "scales"?: [a, b]} — fold two adapters into one (mergeAdapters in src/train/merge.ts; per-source scales default to each adapter’s own config). Output goes to ~/.cache/mlx-bun/adapters/merged-<timestamp>: { "ok": true, "merged_path": "…", "stats": { "layersMerged", "layersOnlyInOne", "totalKeysOut", "sources", "scales" } }; 400 { "ok": false, "error" } on a missing field or merge failure.

POST /api/finetune/export with {"base_model": "…", "adapter_path": "/dir", "method"?: "…"} — writes a manifest.json describing the adapter into ~/.cache/mlx-bun/exports/export-<timestamp>: { "ok": true, "export_path": "…", "manifest": { "version": 1, "base_model", "adapter_path", "method"?, "created_at" } }; 400 on a missing field.

Quantize (/api/quantize/*)

POST /api/quantize/inspect with {"model_id": "<repo id | path>"} — resolve a model through the registry and report whether the quantizer can walk it (inspectModel, src/quantize/job.ts): { "ok": true, "model_id", "arch": "<model_type>" | null, "support": true, "size_gb": 0.0 }, or { "ok": false, …, "error" } (HTTP 200 either way).

POST /api/quantize/submit — starts a quantize job whose output lands in the HF hub cache as a synthetic repo <org>/<name>-OptiQ-<suffix> (refs/main written so the registry picks it up; <suffix> is <bits>bit or mixed-<target_bpw>bpw, plus -rot<seed> when rotating):

{
  "model_id": "org/name | /path",   // required (400 "model_id required")
  "bits": 4, "group_size": 64,
  "target_bpw": 3.5,                // mixed-precision target; wins over bits in the name
  "candidate_bits": [2, 3, 4, 8],
  "reference": "…", "calibration_mix": "…", "n_calibration": 0,
  "rotate_weights": false, "rotation_seed": 42
}

Response: { "ok": true, "job_id": "…", "output_dir": "…" }. The library cache is invalidated when the job completes.

POST /api/quantize/resolve-folder and POST /api/model/resolve-folder (same handler) with {"folder_name"?: "…", "rel_path"?: "…"} — turn a folder the browser picked (a models--org--name hub dir, a snapshot hash dir, or a bare model dir under the hub cache / ~/.cache/mlx-bun) into a servable path: { "ok": true, "path": "…", "repo_id"?: "…", "model_type"?: "…" }, or { "ok": false, "error": "Couldn't locate this folder on disk — paste the path instead." } (HTTP 200).

Dataset (/api/dataset/*)

GET /api/dataset/templates — the dataset-builder template registry (src/dataset/registry.ts): { "templates": [{ "id", "label", "description", "output_format": "messages" | "prompt_completion" | "dpo" | "text", "needs_llm": bool, "fields": [{ "name", "type": "text" | "textarea" | "number", "label", "hint"?, "required"?, "default"? }] }] }.

POST /api/dataset/submit with {"template_id": "…", "inputs": {…}, "model_name"?: "…"} — runs the template in-process as a job; LLM-backed templates call back into this server’s own /v1 (api_url = http://127.0.0.1:<port>). Output dir is ~/.cache/mlx-bun/datasets/dataset-<template>-<timestamp>. Response: { "ok": true, "job_id": "…", "output_dir": "…" }; 400 { "ok": false, "error": "unknown template …" } for an unknown id.

POST /api/{quantize,finetune,dataset}/push

Upload a job’s output (or any folder) to the Hugging Face Hub with the saved write token (src/hf-push.ts). Body: {"repo_id": "org/name", "job_id"?: "…", "source_path"?: "/dir", "private"?: false} — repo_id is required; the source is source_path, else the job’s output_path. The dataset variant pushes as a dataset repo, the others as model repos. Response { "ok": true, "url": "…" }; 400 { "ok": false, "error" } when no token is saved, no source directory resolves, or the upload fails.

GET /api/gc/plan · POST /api/gc/execute

Reclaim disk from superseded snapshots + orphaned blobs in the local HF cache — the web-UI equivalent of mlx-bun gc (thin wrappers over src/registry.ts’s planGc/executeGc; same planner the CLI uses). Every downloaded revision the HF cache keeps a snapshots/<commit> dir for; a snapshot no refs/* points at is superseded and, once no surviving snapshot symlinks to a blob, that blob is dead too. Loopback-served admin routes, no separate auth (same bind as every other /api/* route).

GET /api/gc/plan — read-only (config.json + safetensors index reads only, no tensor bytes, no deletion):

{
  "ok": true,
  "superseded": [{
    "repo_id": "…",
    "prune_snapshots": 1,        // unreferenced, safe to delete
    "skipped_snapshots": 0,      // unreferenced but hold files the kept
                                 //   revision lacks — needs --force via the
                                 //   CLI to actually prune; not deletable here
    "dead_blobs": 3,
    "reclaim_bytes": 0
  }],
  "reclaim_bytes": 0             // sum across every repo
}

POST /api/gc/execute with body {"yes": true} — deletes exactly what a fresh plan would show (recomputed at execute time, not the caller’s stale plan); {"yes": true} is required or the request 400s, mirroring the CLI’s --yes gate so a stray call can’t delete anything by accident:

{ "ok": true, "snapshots": 1, "blobs": 3, "reclaimed_bytes": 0 }

Errors follow the same { "ok": false, "error": "…" } shape as the other /api/* job routes.

GET /api/memory/* · POST /api/memory/init

Thin loopback JSON wrappers over src/memory/vault.ts for the web chat’s Memory panel — same loopback-only admin-route posture as /api/gc/*, no separate auth. Handlers live in src/memory/rest.ts; they never touch the agent-tool surface (src/memory/tools.ts), which stays read-only. Every route below degrades gracefully — a missing vault, an unknown article, or a bad param returns { "ok": false, ... } (with an appropriate HTTP status), never a 500.

When no vault exists yet, every GET route (except /init) returns:

{ "ok": false, "enabled": false, "error": "no memory vault yet", "root": "…" }

GET /api/memory/status — vaultStatus() as JSON:

{ "ok": true, "enabled": true, "status": { "root": "…", "exists": true, "articleCount": 12, "referenceCount": 9, "isGitRepo": true, "recentArticles": [{ "article": "…", "mtimeMs": 0 }] } }

GET /api/memory/list — article stems plus Reference/* doc ids, kept separate (mirrors memory_list):

{ "ok": true, "articles": ["Alpha", "Beta"], "reference": ["Reference/mlx-bun_README"] }

GET /api/memory/search?q=<query>&scope=all|articles|reference&limit=50 — wraps searchArticles (limit capped at 200; 400 without q); same summaries/hits shape the memory_search tool formats from:

{ "ok": true, "summaries": [{ "article": "…", "occurrences": 3, "matched_terms": ["…"] }], "hits": [{ "article": "…", "anchor": "…", "line": 10, "excerpt": "…" }] }

GET /api/memory/article?name=<article> — rendered source plus the deterministic metadata the panel needs (infobox, lead, series banner, section skeleton); 404 { "ok": false } for an unknown name:

{ "ok": true, "name": "…", "path": "…", "content": "# …", "infobox": { "type": "…", "entityKind": "thing", "fields": [] }, "lead": "…", "series": null, "structure": [] }

GET /api/memory/links?name=<article> — inbound/outbound wikilinks (same data memory_links resolves); articles only, not Reference/*:

{ "ok": true, "name": "…", "outbound": ["…"], "inbound": ["…"] }

GET /api/memory/history?name=<article>&limit=50 — the article’s git log (read-only git log plumbing on the vault’s own repo, via Bun.spawn — never a shell string; limit capped at 200):

{ "ok": true, "name": "…", "isGitRepo": true, "entries": [{ "hash": "…", "date": "YYYY-MM-DD", "subject": "…" }] }

GET /api/memory/diff?name=<article>&rev=<hash> — that commit’s diff for the article (git show <rev> -- <path>). rev must match /^[0-9a-f]{4,40}$/ (a bare commit hash — never a ref expression like HEAD~1) or the request 400s before touching git; an unknown name or a revision with no diff for it 404s:

{ "ok": true, "name": "…", "rev": "…", "diff": "diff --git a/articles/…" }

POST /api/memory/init with optional body {"path"?: string} — the first-run consent-card backend. Delegates to the same setupVault() the CLI’s mlx-bun memory init calls (idempotent: create dirs, write README + Meta pages only if missing, git init + initial commit); omits the CLI’s interactive extras. path, if given, must resolve under the default vault root or the OS temp directory, else 400:

{ "ok": true, "result": { "root": "…", "created": ["…"], "gitInitialized": true, "alreadySetUp": false }, "status": { "root": "…", "exists": true, "articleCount": 0, "referenceCount": 9, "isGitRepo": true, "recentArticles": [] } }

GET /v1/memory/synthesize

Runs the memory synthesis pipeline (runSynthesis, src/memory/pipeline.ts — the same job mlx-bun memory synthesize runs) and streams its progress as SSE. ?dry=1 plans the stages without model calls or writes. This is a long GPU job over the whole store on a real run; it is registered under /v1/ for historical reasons and is not part of the OpenAI surface.

data: {"type":"stage","stage":"ingest","message":"…"}
data: {"type":"log","message":"…"}
data: {"type":"done","message":"synthesis done — …"}
data: {"type":"summary","implemented":true,"stages":["ingest","segment","extract","route","create","section-route","patch","link","wikify"],"note":"…"}
data: [DONE]

A failure emits data: {"type":"error","message":"…"} and closes without [DONE].

Settings (/api/settings/*)

GET /api/settings/hf-token → { "ok": true, "hasToken": bool }; POST /api/settings/hf-token with {"token": "hf_…"} saves the Hugging Face write token to ~/.mlx-bun/hf.json (mode 0600) → { "ok": true }; 400 "token required" when empty. The token drives the push routes above and the upload CLI command (cli.md).

GET /api/settings/tool-approvals / DELETE /api/settings/tool-approvals — the web chat’s durable “always allow this tool” list for gated tools (bash/edit/write), backed by ~/.mlx-bun/tool-approvals.json (src/tool-approvals.ts), keyed by stable tool NAME, not per-call arguments. These two routes are read/forget only: granting an always-allow happens through the approval card itself (the approval WS frame on /ws/chat with alwaysAllow: true), never a bare settings POST.

// GET
{ "ok": true, "alwaysAllow": ["bash", "edit"] }
// DELETE with body {"tool": "bash"} — idempotent; returns the updated set
{ "ok": true, "alwaysAllow": ["edit"] }

DELETE without tool is 400 "tool required".

Model Hub (/api/hub/*)

The web chat’s Model Hub panel: browse downloaded models, search Hugging Face, and kick off downloads — all loopback-served, no separate auth. Handlers live in src/hub-rest.ts (pure functions, no loaded-model dependency).

GET /api/hub/local — every model in the local registry (same source /library reads), each with a /fit-computed hardware verdict at a fixed 8k context:

{
  "ok": true,
  "models": [{
    "repo_id": "…",
    "model_type": "gemma3" | "minicpm5" | "qwen3" | …,
    "size_bytes": 0,
    "quant_bits": 4,
    "quant_group_size": 64,
    "vision": false,
    "supported": true,
    "support_tier": "targeted" | "generic" | null,
    "assessment": { "fits": true, "max_safe_context": 8192, "predicted_decode_tps": 0.0 } | null
  }]
}

GET /api/hub/search?q=<query> — server-side Hugging Face model search, filtered to the mlx library tag so results are MLX-compatible by construction. Never downloads anything. 400 without q. Degrades on any network failure (DNS, timeout, non-2xx) to an explicit offline state rather than a 500:

{ "ok": true, "offline": false, "results": [{ "id": "mlx-community/…", "downloads": 0, "likes": 0, "size_estimate": null }] }

{ "ok": true, "offline": true, "error": "…", "results": [] }

size_estimate is always null — the search endpoint doesn’t return per-repo file sizes; the fit badge lands once a model is actually downloaded (/api/hub/local).

POST /api/hub/download with body {"repo": "org/name"} — starts a background download via downloadModel() and returns immediately; progress is visible via GET /downloads. Refuses a duplicate kick-off for a repo that’s already downloading:

{ "ok": true, "repo": "org/name", "started": true }

{ "ok": false, "error": "a download for org/name is already in progress" }   // 409

POST /api/hub/serve with body {"model": "org/name"} — always answers restart_required, never performs a live in-process swap. The process a web chat session is attached to has exactly one model loaded and no drop-weights-and-reload seam reachable from /ws/chat (the --isolate proxy’s ModelPool is a real spawn-overlap swap, but it lives on the isolated /v1/* surface only — src/serve/isolate.ts). The honest answer is the restart command (HTTP 200, ok: false):

{ "ok": false, "restart_required": true, "command": "mlx-bun serve org/name" }

GET /api/sessions/search · GET /api/sessions/export

Full-text search across web-chat session message bodies and chat export. Same loopback-only, no-separate-auth posture as /api/memory/*. Handlers live in src/serve/session-search.ts (pure, read-only, no loaded-model dependency). Both routes read the same session directory the web chat’s PiWebSession writes to (~/.mlx-bun/sessions, pi’s own JSONL-per-session format) and are JSONL-tolerant — a corrupt or mid-write line is skipped, never thrown.

GET /api/sessions/search?q=<query> — case-insensitive substring scan (no index — personal-chat-corpus scale) over every session file’s user/assistant message text, capped at 50 matches total (10 per session). Each match reports a plain-text ±60-char snippet plus the match’s own [start,end) offsets into that snippet — offsets, not pre-rendered HTML, so the frontend escapes the snippet then inserts its own <mark>:

{
  "ok": true,
  "results": [{
    "sessionPath": "/Users/…/.mlx-bun/sessions/20260706_…jsonl",
    "sessionTitle": "Chicken recipes",
    "matches": [{ "snippet": "…tell me about rosemary and thyme…", "ranges": [[14, 22]], "role": "user" }]
  }]
}

400s {"ok": false, "error": "q is required"} without a query.

GET /api/sessions/export?path=<session file> — the raw session JSONL, parsed into an array of entries (one per line, in file order). path must resolve under the session directory:

{ "ok": true, "path": "…", "entries": [{ "type": "session", "id": "…", "cwd": "…" }, { "type": "message", "message": { "role": "user", "content": "…" } }] }

400s without path; 403s a path outside the session directory; 404s a path inside the directory that doesn’t exist on disk.

Web app static routes + PWA installability

The web chat (GET /) and everything it loads same-origin, no CDN, ever (src/serve/static-routes.ts; GET only):

Route	Content	Notes
GET /	src/web/app.html	The unified SPA shell.
GET /assets/app.js	src/web/app.js	GENERATED from src/web/src/*.ts by bun scripts/build-web.ts — see tests/using/web-build.test.ts’s freshness gate.
GET /assets/hljs.js, GET /assets/hljs.css	src/web/vendor/hljs*	Vendored syntax highlighting, no CDN.
GET /manifest.webmanifest	src/web/manifest.webmanifest	PWA manifest.
GET /assets/icon.svg	src/web/icon.svg	A single inline SVG app icon; no binary PNGs are shipped.
GET /sw.js	src/web/sw.js	Service worker, served cache-control: no-store at the root scope.
GET /curves	src/lab/curve/curve-designer.html	The HLG Curve Designer page (drives /signal + /generate below); read from disk when available, embedded copy in compiled binaries. no-store.
GET /curve-terrain	src/serve/assets/curve-terrain.html	Curve terrain visualization; 404 when the artifact isn’t on disk.
GET /dag	src/serve/assets/training-inference-map.html	Training/inference DAG map; 404 when the artifact isn’t on disk.
GET /status, /chat, /quantize, /finetune, /dataset	—	Legacy page paths: 302 to /#<path> (the SPA’s hash routes).

/assets/* and the manifest are served with cache-control: public, max-age=3600; /sw.js, /curves, /curve-terrain, /dag are no-store.

PWA scope: the manifest + service worker exist for installability and an instant static-shell paint — “Add to Home Screen” / a browser “Install app” prompt, and the next visit’s HTML/JS/CSS loading from cache while the WebSocket connects. This is explicitly not offline chat: src/web/sw.js’s fetch handler only intercepts the exact shell files (/, /assets/app.js, /assets/hljs.js, /assets/hljs.css) and lets every dynamic route (/api/*, /v1/*, /ws/chat, /downloads, …) fall straight through to the network unconditionally. Registration is guarded to secure contexts (https: or localhost/127.0.0.1).

Curve Designer: POST /signal · POST /generate

Backends for the /curves page (HLG tone-curve sampling research tool, src/lab/curve/curve-sampler.ts). Both answer OPTIONS and send permissive CORS headers (access-control-allow-origin: *) so the page can be opened from a file or another origin. Both take the GPU lock (gateway.runExclusive) — they never interleave with a serving request.

POST /signal with {"prompt": "…"} — one prefill forward of the prompt rendered as a single user turn; bins the next-token log-prob distribution into 80 buckets over the curve’s x-axis so the editor can draw the signal under the curve: { "bins": [80 ints], "vocab": V }. 400 {"error": "invalid JSON"}; 500 {"error": "signal failed: …"}.

POST /generate with {"prompt", "curve"?: {points: [...]}, "n"?: 3, "max_tokens"?: 80, "seed"?: 0, "default"?: false} — samples n (1–8) completions of up to max_tokens (1–256) tokens. When curve has ≥2 points and default is not true, the drawn log-prob transfer curve replaces temperature+softmax entirely; otherwise the model’s default chat recipe (temperature/top-p/top-k) is used. A non-monotone curve is a 400.

{ "mode": "curve" | "default",
  "recipe": { "temperature": 0.7, "topP": 0, "topK": 0 },   // only in default mode
  "n": 3, "seed": 0,
  "samples": [{ "text": "…", "junk": false }] }             // junk = coherence heuristic tripped

Web chat WebSocket (/ws/chat)

GET /ws/chat upgrades to a WebSocket (426 when the upgrade header is missing; 501 under --isolate). It carries the web chat’s pi AgentSession — the model is driven through pi, which calls back into this server’s own loopback /v1. Frames are JSON objects with a type field; the exact TypeScript unions are ClientMessage and ServerMessage in src/pi-web.ts (the web frontend imports them type-only, so both ends are compile-checked).

Client → server: prompt {text, images?: [{data, mimeType}]}, abort, approval {callId, decision, editedArgs?, alwaysAllow?}, set_thinking {enabled}, set_coding_tools {enabled}, set_adapter {id | null}, set_sampling {scope?: "session" | "next_turn", temperature?, top_p?, top_k?, min_p?, xtc_probability?, xtc_threshold?, repetition_penalty?, repetition_context_size?, presence_penalty?, frequency_penalty?, seed?} (null = server default), set_system_prompt {text | null}, new_session, list_sessions, open_session {path}, fork_session {path}, delete_session {path}, regenerate, edit_resend {text}, switch_sibling {entryId}, context {context} (app-aware assistant, below).

Server → client: ready {model, vision, audio, thinking, genDefaults, transcription} (transcription is true when a Whisper checkpoint serves /v1/audio/sessions; the web composer then shows a hold-to-talk mic that streams 16 kHz float32 into a session and inserts the transcript into the message box — voice input for any chat model), turn_start, text_delta {delta}, thinking_delta {delta}, tool_start {callId, tool, args}, tool_approval_request {callId, tool, args}, tool_update {callId, chunk}, tool_end {callId, ok, result}, turn_end {lane?} (the serving lane the turn ran on, from usage.lane via the lane registry — never guessed client-side), queue_update {steering, followUp}, history {items}, sessions {items, activePath?}, siblings {entryId?, index, count, siblingIds}, context {tokens, contextWindow, percent} (context-window usage — a different frame from the client→server context), coding_tools {active, pending}, tool_approvals {alwaysAllow}, ui_navigate {route}, ui_spotlight {ref?, label?, selector?, target?, message?, route?}, error {message}.

App-aware assistant

The web chat agent can see and act on the app it lives in. No screenshots and no vision model are involved: the browser sends a structured DOM snapshot as a WS frame, and the model gets three tools that read/navigate/highlight that snapshot. The three tools are always in the web chat’s tool allowlist (read-only-on-the-machine, never gated by the approval card; APP_AWARE_TOOL_NAMES in src/pi-web.ts):

• get_current_app_context — returns the last context the browser pushed: { route, view?, step?, snapshot }. snapshot is a capped (~120) list of visible interactive elements as {ref, label, kind, role?, selector, spotlightId?}, agent chrome excluded.
• navigate_app({ route | page }) — validates against the route catalog (APP_ROUTE_IDS: chat, quantize, finetune, dataset, status; unknown routes are a tool error, never sent to the browser) and emits a ui_navigate frame.
• spotlight_ui({ ref | label | selector | target, route?, message? }) — emits a ui_spotlight frame; the browser resolves the target (ref from its last snapshot, then live selector, then fuzzy label match, then a curated catalog id) and shows a brief non-blocking highlight.

Client → server WS frame (pushed by the browser on every route change and every wizard-step change, never on a timer):

{ "type": "context", "context": { "route": "quantize", "step": { "index": 1, "count": 4, "label": "Configure" }, "snapshot": { "route": "quantize", "capturedAt": "…", "elements": [/* … */] } } }

Server → client frames the two action tools produce:

{ "type": "ui_navigate", "route": "quantize" }
{ "type": "ui_spotlight", "target": "quantize-source", "message": "Paste a Hugging Face repo id or local path here" }

The server also auto-prepends a compact one-line ambient context (e.g. [user is on: Quantize · step 2/4]) to the system prompt on every turn — never a full snapshot dump. Approval-gated form-fill / job-start actions are not wired; only navigate/spotlight are.

Client setup: pi

The supported one-command path:

mlx-bun harness pi            # install; then: pi --provider mlx-bun
mlx-bun harness pi --remove   # undo

It writes a self-contained extension to ~/.pi/agent/extensions/mlx-bun-provider.ts that registers the mlx-bun provider with live discovery: at pi startup it fetches /v1/models from the running server, so the stable mlx-bun/local model handle always resolves to whatever the server is actually serving (context window, reasoning, and vision capability included), and never goes stale across model swaps. The model list baked at install time is only the fallback for when the server is down.

Manual fallback — ~/.pi/agent/models.json:

{
  "providers": {
    "mlx-bun": {
      "baseUrl": "http://127.0.0.1:8080/v1",
      "api": "openai-completions",
      "apiKey": "sk-anything-nonempty",
      "models": [{ "id": "<model id from /v1/models>" }]
    }
  }
}

Note the manual route bakes a concrete model id, which goes stale when you serve a different model — the staleness harness pi’s local id exists to solve.

Any OpenAI SDK works the same way: baseURL: "http://127.0.0.1:8080/v1", any non-empty apiKey.

Under --isolate, the parent owns Responses continuation history and exposes its bounded store counters in GET /engine as response_store. Model-worker restart or eviction preserves that history; restarting the parent clears it. Direct serving retains the same process-local TTL and byte limits.

Supported grouped TurboQuant drafting retains usage.speculation, seeded sampling, logprobs, structured output and generated RAM/SSD continuations. The cache codec does not select the scheduler or sampler implementation. Composition coverage: batching design.

With paged KV enabled, supported Gemma4 bf16 text requests report the batched lane when submitted through continuous scheduling. Seeded sampling, logprobs and grammar use the same interfaces. These requests bypass prompt-cache lookup/publication; paging does not provide SSD persistence.

Ordinary adapter checkpoints are isolated by ordered adapter IDs and mounted content revisions. Base-model requests cannot restore or remove an adapter checkpoint. With generation checkpointing enabled, repeating an identical eligible request replays its saved completion prefix and resumes from the saved pending token. Cancellation retains queued snapshots; successful completion removes them. POST /admin/cache/flush drains both prompt spills and generation checkpoints, and its elapsed time and durability statistics include both queues. A hard process exit can lose snapshots that have not drained.

For resumed generation checkpoints, usage.prompt_tokens_details.cached_tokens counts only the reused original prompt, bounded by usage.prompt_tokens. Saved completion tokens are replayed as completion output and do not increase that prompt-cache usage counter.