Prompt Caching Explained (2026): How It Works, OpenAI vs Anthropic vs vLLM

Prompt caching is an LLM inference optimization that stores the computed state (the KV cache) of a prompt’s static prefix — your system prompt, tool definitions and documents — so repeat requests skip recomputing it. The result: up to ~85% lower time-to-first-token and up to ~90% lower cost on cached input tokens, without changing your model or your outputs. I use it in production on a RAG assistant; here is how it works, what OpenAI, Anthropic and vLLM each do differently, and the mistakes that silently kill your cache-hit rate.

What is prompt caching?

Every LLM request goes through two phases. Prefill processes your whole prompt in parallel — tokenizing it, running attention, and building the KV cache (the key/value tensors every layer computes for every token). Decode then generates the answer token by token. Prefill is compute-bound and dominates time-to-first-token; decode always runs.

In agents and RAG assistants, 90%+ of every request is an identical prefix: the same system prompt, the same tool definitions, the same knowledge base — only the user’s question changes. Without caching, the model recomputes all of that on every call. Prompt caching stores the prefix’s KV cache and reuses it, so the model only processes the new tokens. It accelerates prefill only — generation still happens, which is why your outputs are unchanged.

How the providers compare (2026)

OpenAI: automatic, ≥1,024 tokens

OpenAI’s prompt caching is automatic — no code changes, no extra fee. It activates for prompts of 1,024 tokens or longer, and cached input is discounted up to 90% on current models (the widely-quoted 50% figure is the stale 2024 launch number). Cached prefixes stay warm for roughly 5–10 minutes of inactivity, and you can improve routing for shared prefixes with the prompt_cache_key parameter. Check usage.prompt_tokens_details.cached_tokens to see it working.

Anthropic (Claude): explicit breakpoints, biggest discounts

Claude’s caching is explicit: you mark up to 4 cache_control breakpoints across tools, system and messages. Cache reads bill at 0.1× the input price (90% off). Writes carry a surcharge — and here is the nuance most posts miss: 1.25× for the default 5-minute TTL, but 2× for the 1-hour TTL. So a 5-minute cache breaks even on the second request, while the 1-hour cache needs three or more reads to pay for itself. The minimum cacheable prefix is model-dependent (roughly 1,024–4,096 tokens) — shorter prefixes silently don’t cache, no error raised. Watch cache_creation_input_tokens and cache_read_input_tokens in the usage block.

vLLM (self-hosted): automatic prefix caching

If you run your own models, vLLM’s automatic prefix caching is enabled by default in the V1 engine. It hashes KV blocks (each block keyed on its parent’s hash plus its exact tokens) on top of PagedAttention, evicts LRU, and — like all prompt caching — speeds up prefill only. There’s no per-token discount because you own the GPU economics: the win is throughput and latency.

Prompt caching vs semantic caching

These get confused constantly, and they solve different problems. Prompt caching works inside inference: it matches the exact token prefix and still executes the model — it just skips recomputing the prefix. Semantic caching (Redis LangCache, GPTCache) works at the application layer: it embeds incoming questions, matches them against previous ones by cosine similarity, and returns the stored response — skipping the LLM call entirely, even for paraphrased questions. That makes it dramatically cheaper on hits, at the risk of serving a stale or subtly-wrong answer if your similarity threshold is loose. In practice they stack: semantic cache in front, prompt cache behind it.

How to actually get cache hits (from production)

Prompt caching is exact-prefix matching — one changed byte breaks everything after it. The practices that matter:

• Static content first, dynamic input last. System prompt, tool definitions and documents go at the top; the user’s question goes at the end. If you inject a timestamp or a random ID into your system prompt, you just disabled your cache.
• Keep prompts byte-stable. Serialize tool definitions deterministically. In my RAG assistant, the system prompt, tool schemas and retrieved-document scaffolding form the cached prefix; only the retrieval results and question vary.
• Version system prompts deliberately. Every edit is a full cache invalidation — batch your prompt changes instead of tweaking constantly.
• Cache the big boring things. Long documents and large tool catalogs are exactly where the 90% discount compounds.
• Measure cache-hit ratio like a product metric. Both OpenAI and Anthropic report cached tokens per request — alert on regressions, because a silent drop in hit rate is a silent cost increase.

Does prompt caching actually save money?

Do the math per prefix, not per marketing page. On Claude with the 5-minute TTL: a cached prefix costs 1.25× once, then 0.1× per read — break-even on the second request, and ~87% savings on prefix input by the tenth. A chatbot with steady traffic keeps the 5-minute window refreshed indefinitely (each read resets the timer). Low-traffic or bursty workloads may need the 1-hour TTL — but at 2× write cost, confirm you’ll get 3+ reads. On OpenAI it’s simpler: caching is automatic and free, so structure your prompts for it and the discount just shows up.

Prompt caching FAQ

Does prompt caching change the model’s answers?

No. The cached KV tensors are exactly what the model would have computed — generation runs identically. It is a performance optimization, not a behavior change.

Is prompt caching free?

On OpenAI, yes — automatic, discount included. On Anthropic, reads are 90% off but writes carry a 25% (5-min) or 100% (1-hour) surcharge, so it pays for itself from the second request. On vLLM you pay nothing extra — it’s your hardware.

Why is my cache hit rate low?

Almost always: something dynamic (timestamp, user ID, shuffled tool order) sits early in your prompt and breaks the prefix, or your prefix is below the provider’s minimum cacheable length.

I build agentic AI and RAG systems in production — where optimizations like this are the difference between a demo and a product. If you’re working on LLM infrastructure or hiring engineers who obsess over these details, let’s talk.