Queueing

Updated 2026-10-30.

A request for a model that is not loaded waits in the GPU queue while the GPU runs an image or music job.

Queue

GPU state	Result
Requested model loaded	Runs at once; queued behind an image or music job that has waited 300 s
GPU idle, or another model loaded	Model switch starts
GPU running an image or music job	Waits in the queue up to 900 s, then switches
GPU unavailable for 900 s	`503 gpu-busy`

A loaded model with requests in progress stays loaded. After 90 s without requests, an image or music job can unload the model. A job that has waited 300 s behind a steady stream of requests goes next: new requests for the loaded model wait in the queue behind it, then the model loads again.

Request	Result while queued
Streamed	Waits; `: loading` comments every 15 s
Not streamed, no `X-LLM-Wait`	Retried by Citadel for 80 s, then `503 queued`
`X-LLM-Wait: 0`	`503 queued` at once, with `position`

Retries retain the queue position.

503 queued, Retry-After: 30

{
  "error": {
    "message": "Waiting for the GPU (1 ahead); retry in 30 s",
    "type": "queued",
    "code": "queued",
    "model": "qwen3.8-27b",
    "position": 1,
    "retry_after_s": 30,
    "reason": "waiting for the GPU: a media job is running"
  }
}

503 gpu-busy, Retry-After: 30

{
  "error": {
    "message": "The GPU is busy (a media job is running); retry in 30 s",
    "type": "gpu-busy",
    "code": "gpu-busy",
    "reason": "a media job is running"
  }
}

Field	Value
`position`	Requests ahead in the queue
`retry_after_s`	Same value as `Retry-After`
`reason`	Human-readable text; varies

Branch on code. Retry queued, gpu-busy and model_loading after Retry-After seconds.

Concurrency

qwen3.8-27b runs up to 64 requests at once. Requests above 64 wait in the engine. Total throughput rises with concurrency. Per-request speed falls.

Concurrent requests	Total tok/s	Per request tok/s
1	41–42	41–42
8	142–204	32–34
30	249–359	14–17

Measured 2026-10-29: qwen3.8-27b, streamed, thinking off, 200 output tokens, about 940-token prompts. The higher total is the second turn, with prefix cache reuse.

Open LLM calls are limited to 96 across all keys. Call 97 returns 503 device_busy with Retry-After: 1.