Prefix caching

Prefix caching reuses the computed state (KV cache) of a prompt prefix from an earlier request. Cached prompt tokens skip prefill, which shortens the time to the first token.

Example

usage.prompt_tokens_details.cached_tokens reports the cached prompt tokens.

import os
from openai import OpenAI

client = OpenAI(base_url="https://citadel.albastov.com/v1", api_key=os.environ["CITADEL_API_KEY"])
no_thinking = {"chat_template_kwargs": {"enable_thinking": False}}
history = [
    {"role": "system", "content": "You are a terse assistant. " + "Answer in at most two sentences. " * 40},
    {"role": "user", "content": "What is a KV cache?"},
]
first = client.chat.completions.create(model="qwen3.8-27b", messages=history, max_tokens=80, extra_body=no_thinking)
history += [{"role": "assistant", "content": first.choices[0].message.content},
            {"role": "user", "content": "And prefix caching?"}]
second = client.chat.completions.create(model="qwen3.8-27b", messages=history, max_tokens=80, extra_body=no_thinking)
for name, r in (("turn 1", first), ("turn 2", second)):
    print(name, r.usage.prompt_tokens, "prompt tokens,", r.usage.prompt_tokens_details.cached_tokens, "cached")

Measurements

Measured 2026-10-29: qwen3.8-27b, about 940-token prompts, all conversations sent at once, medians.

Cache reuse

• Resend earlier messages unchanged: same order, same text. Append new turns at the end.
• Put the fixed system prompt first. Put per-request content (dates, retrieved documents) after the fixed part.
• Send earlier assistant turns without reasoning.
• A model switch clears the cache.

Cache size

The qwen3.8-27b cache is about 6.35 GiB, for all callers. A full cache evicts the oldest entries.

30 conversations of about 2,400 tokens fill about 78 % of the cache. In that test, turn 2 found 240 cached tokens per conversation, and time to the first token rose from 23.0 s to 27.4 s. Run long conversations in groups of 8 to 10.

Not measured: qwen3.8-27b-fast, qwen3.8-flash-next-abliterated, deepseek-v4.1-flash, the llama.cpp model.