Prompt caching

Updated

Prompt caching keeps repeated long prefixes (system prompts, documents, tool definitions) from being charged at full price every time. Cached tokens are billed at the cache read price, usually far below the input price.

How to use it

  • Anthropic protocol: add "cache_control": {"type": "ephemeral"} to the content blocks to cache, exactly as in the official API.
  • OpenAI and Gemini protocols: most upstreams cache long repeated prefixes automatically, with no extra parameter.
{
  "model": "MODEL_ID",
  "max_tokens": 1024,
  "system": [
    {"type": "text", "text": "A very long product manual goes here...", "cache_control": {"type": "ephemeral"}}
  ],
  "messages": [{"role": "user", "content": "What is the return policy?"}]
}

How the platform handles it

  • Whether an endpoint supports prompt caching is shown by the capability tags on the model page. Endpoints without caching still serve the request; they just never hit the cache, so cache_control never makes a request fail.
  • The cache lives with the provider, so a different endpoint means rebuilding it. Send an X-ModelPortal-Session header with multi-turn conversations and the platform tries to keep a session on one endpoint; see Default routing.
  • prompt_cache_key can identify users, so the platform removes it before forwarding.
  • Cache reads and cache writes are priced separately in usage. Endpoints without a separate cache price bill cached tokens at the input price; see Billing.