LLM Caching: Exact, Prefix, and Semantic, and What Each Actually Saves

Caching is the second-highest-leverage inference cost lever after model routing, and the three cache types are not variations on one idea: they solve different problems, deliver different savings, and fail in different ways. Provider-side prefix caching cuts the cost of context you send repeatedly, is nearly free to adopt, and should be turned on almost everywhere. Exact-match response caching only fires on byte-identical requests, and most teams overestimate its hit rate by an order of magnitude. Semantic caching, which looks up an embedding-similar earlier request before generating, is where the real hit rates live on user-facing traffic and also where the correctness risk lives, because a near-miss served confidently is worse than a miss.

The conflation is the expensive part. Teams that hear "caching" and reach for one implementation usually reach for exact match, get a hit rate in the low single digits, conclude caching does not work for their workload, and leave prefix caching switched off, which would have cut the bill immediately at no correctness risk.

Key Takeaways

• The three cache types are complements, not alternatives. Prefix caching cuts the cost of repeated input tokens, exact caching eliminates repeated identical work, and semantic caching eliminates repeated equivalent work. A mature stack runs all three at different layers.
• Turn on prefix caching first. It requires structuring the prompt so the stable part comes first, carries no correctness risk because the model still generates the answer, and pays back immediately on any workload with a long system prompt, few-shot examples, or a document held across turns.
• Exact-match hit rates are far lower than teams predict. Free-text user input is effectively never byte-identical, so exact caching earns its place on machine-generated and enumerable requests rather than on conversational traffic.
• Semantic caching is the only one that can be wrong. It trades correctness for hit rate through a similarity threshold, and it fails hardest on negation, entities, and numbers, where two requests are textually near-identical and semantically opposite.
• Measure savings as hit rate multiplied by cost per miss, not hit rate alone. A ninety percent hit rate on the cheapest requests in the workload can be worth less than a fifteen percent hit rate on the expensive tail.

Why Caching Ranks Second

Every inference cost program works through the same short list: send fewer tokens, reuse work, change the price tier, or negotiate the rate. Routing ranks first because it moves a request across a price boundary rather than shrinking it, an argument developed in the guide to LLM model routing in production. Caching ranks second because it removes the work entirely, and that beats doing it cheaply.

Caching is underused relative to routing because its value is unevenly distributed and easy to measure wrong. Routing savings are broad and predictable; caching savings depend on the repetition structure of the specific traffic, which most teams have never measured. The first useful step is not implementing a cache but instrumenting the traffic to find what repeats: identical requests, similar requests, and identical prefixes are three different measurements, each pointing at a different cache.

The Three Cache Types

Prefix Caching: Cutting the Cost of Repeated Context

Prefix caching is a provider-side mechanism that stores the model's computation over the leading portion of a prompt so the same prefix is not reprocessed on the next request. It is billed as a discount on input tokens that hit the cache, and it is the only one of the three operating inside the model provider rather than your infrastructure.

The mechanics impose one hard constraint: the cache matches on an exact prefix, so anything variable must come after everything stable. That ordering rule is most of the implementation work. System prompt, tool definitions, few-shot examples, and long reference documents belong at the front, in fixed order, byte-identical between requests. The user's query, per-request retrieved passages, and the timestamp go at the end. Teams that interpolate a session identifier or current date near the top destroy the cache for every subsequent request and usually do not notice, because nothing breaks, the bill just fails to fall.

The economics favor the right shape of workload. Providers price cached input tokens well below uncached ones, and savings scale with how much of the prompt is stable and how often it repeats. An agent with a long tool-definition block, a document-analysis flow holding one document across many questions, or any application with substantial few-shot examples will see a large share of input tokens hit the cache. Workloads with short prompts and long completions gain little, because the cost sits in output tokens no prefix cache touches.

Two operational details matter. Entries expire, with provider-dependent lifetimes typically measured in minutes, so sparse traffic may never hit a warm cache while steady traffic hits it almost always. And some providers require a minimum prefix length before caching engages, so short prompts fall through silently. Both are reasons to verify by reading the cached-token counts in the API response rather than assuming.

The best property of prefix caching is that it is correctness-neutral. The model still generates the answer; only the redundant computation over the prefix is skipped. There is no stale-answer risk and no similarity threshold to tune, which is why it should be adopted before either of the other two.

Exact-Match Caching: Real, Narrow, Routinely Overestimated

Exact-match caching stores the response to a request and returns it when a byte-identical request arrives, typically keyed on a hash of the full prompt plus the parameters that affect generation.

Cache-key design is where correctness is won or lost. The key must include everything that changes the answer: the full prompt text, the model identifier and version, temperature and other sampling parameters, the system prompt, any retrieved context injected, and the response schema where one is enforced, since a change to the structured output schema changes the response shape while leaving the prompt untouched. Omitting the model version is the classic error, because it means a model upgrade silently continues serving answers produced by the previous model. The key must also include a tenant or user scope wherever the response could contain data specific to that tenant, since a cache shared across tenants without scoping is a data-leak mechanism rather than an optimization.

The honest expectation on hit rate is what teams get wrong. Free-text human input is essentially never byte-identical: users phrase the same question differently, add or omit punctuation, vary capitalization. Conversational requests also carry accumulated history, making each unique by construction. Low single-digit hit rates are normal for this traffic, and a team forecasting thirty percent is forecasting a workload it does not have.

Exact matching earns its place on machine-generated and enumerable requests: a classification call fired against the same document by multiple pipeline stages, a batch job reprocessing overlapping inputs, evaluation runs replaying a fixed suite, and product surfaces with a bounded set of canned queries. Normalization helps at the edges. Trimming whitespace, lowercasing where it does not change meaning, and canonicalizing key ordering will lift the rate somewhat, and each step is a small correctness bet that two textually different requests deserve the same answer.

Semantic Caching: Where the Hit Rates and the Risk Both Live

Semantic caching embeds the incoming request, searches a vector store for a previously answered request above a similarity threshold, and returns the stored response on a hit. It is the only cache type that can return a wrong answer, and it is also the only one that produces meaningful hit rates on natural language traffic.

The threshold is the entire design. Set it high and the cache behaves like a slightly more forgiving exact match: low hit rate, few errors. Set it low and hit rate climbs while the cache starts confidently answering questions nobody asked. There is no universally correct value; it depends on the embedding model, the domain, and how much a wrong answer costs relative to a miss. The only defensible way to choose it is empirically, sampling production request pairs at candidate thresholds and having a judge or a human decide whether the stored answer was acceptable for the new request.

The failure modes cluster in three places, and all three share a structure: the requests are textually close and semantically opposite.

Negation is the sharpest. "Is this transaction eligible for a refund" and "Is this transaction not eligible for a refund" are near-identical in embedding space and demand opposite answers. Most general-purpose embedding models represent negation weakly, which makes this a systematic vulnerability rather than an occasional accident.

Entities and numbers are the second. "What is the balance on account 4471" and "What is the balance on account 4472" differ by one character and mean entirely different things. Any request whose meaning turns on a specific identifier, amount, or date should be excluded from semantic caching, not merely thresholded, because no threshold reliably separates them.

Time sensitivity is the third. "What is our current inventory" was a correct answer when it was cached and is a wrong answer an hour later. Semantic caching has no concept of freshness unless one is imposed through a time-to-live, and choosing that lifetime requires knowing how fast the underlying answer decays, which varies per query type rather than globally.

The mitigations are ordinary: exclude categories rather than trusting the threshold to catch them, scope caches per tenant so a hit cannot cross a customer boundary, set aggressive time-to-live on anything touching mutable state, and invalidate whenever the model version, system prompt, or retrieval corpus changes, since all three change what the correct answer is.

Cache poisoning deserves attention because it is the one failure with an adversary behind it. If user-supplied content becomes a cache entry, a user who can predict or influence what gets cached can plant a response later served to someone else. Tenant scoping closes most of this; writing to the cache only from trusted generation paths closes most of the rest.

Comparing the Three

The ordering advice follows from the risk column. Adopt prefix caching first because it is free of correctness risk. Add exact matching second, where the traffic shape justifies it. Add semantic caching last, and only with an evaluation harness already in place, because it is the only one that can silently damage output quality.

Where Caching Belongs in the Stack

Caching is cross-cutting and belongs at the layer that already sees every request, which for most organizations is the gateway. Per-application caches produce divergent key schemes, inconsistent invalidation, per-team correctness decisions, and no aggregate view of what the program is saving. The consolidation argument for putting routing, caching, rate limiting, and observability behind one control point is developed in the enterprise guide to the AI gateway.

Prefix caching is the exception, since it is enabled by how the application constructs the prompt rather than by anything the gateway does. The gateway's role there is to enforce and verify: flag prompts whose variable content precedes their stable content, and expose cached-token ratios per route so a regression is visible.

One ordering interaction matters. Caching should sit in front of routing, not behind it, because a cache hit means no model is called at all and the routing decision is moot. Running the router first wastes the classification work on every hit.

Measuring What Matters

The metric that misleads is hit rate alone. The metric that decides is hit rate multiplied by the cost of the request that was avoided.

Cache value equals hit rate times cost per miss. A ninety percent hit rate on trivially cheap classification calls can be worth less than a fifteen percent hit rate on long-context retrieval-augmented requests that cost a hundred times more. Segment the measurement by request class or the aggregate will point at the wrong optimization, and the same holds for the token-level accounting that determines the cost per miss in the first place, laid out in the analysis of the hidden costs in inference token pricing.

Four numbers are worth a dashboard. Cached-token ratio per route, the direct read on whether prefix caching is working and the first thing to break when someone edits a template. Hit rate segmented by request class, for exact and semantic caches separately, because blending them hides which is doing the work. Cost avoided per day, hits multiplied by the measured cost of the equivalent miss, which is the number that justifies the program. And for semantic caching only, a correctness rate: the share of sampled hits where the served response was actually acceptable.

That last number is not optional. A semantic cache without a correctness measurement is an unmonitored quality regression, and the sampling needed is modest: a periodic judge pass over a random sample of hits, disputed cases reviewed by a human, is enough to catch a threshold that has drifted.

Latency deserves separate accounting because the wins land differently. Exact and semantic hits remove the entire generation, usually the dominant latency component. Prefix caching mainly improves time to first token, which matters for streaming interfaces and much less for batch work.

Failure Modes to Instrument

Four failures recur, and three of them are silent.

The stale answer after a change is the most common and most damaging. A prompt is edited, a model upgraded, or the corpus reindexed, and the cache keeps serving responses produced under the old configuration. The fix is structural, not procedural: put the model version, prompt template version, and corpus version in the key so any change invalidates affected entries automatically. Relying on someone remembering to flush during a deploy is relying on the thing that will eventually not happen.

The silent prefix-cache regression is the second. Someone adds a timestamp or a request identifier near the top of a prompt template, the prefix stops matching, and costs rise with no error anywhere. Only the cached-token ratio reveals it, which is why that metric belongs on a dashboard with an alert rather than in an occasional review.

Unscoped multi-tenant hits are the third and the most severe when they occur, because a cache that can serve one customer's response to another is a data incident. Tenant scoping belongs in the key, enforced at the gateway rather than left to each application.

The fourth is the semantic threshold that was tuned once. Embedding models get updated, traffic distribution shifts, and a threshold calibrated at launch drifts out of alignment without any signal other than the correctness sampling described above.

A Build Order

Stage four before stage five is the load-bearing part of that ordering. Shipping semantic caching without the harness means the threshold is a guess and the error rate unknown, and its errors are exactly the kind that surface weeks later as a quality complaint nobody can trace. As of the week of September 7, 2026, tooling for this is adequate but not turnkey, which is the honest reason many teams stop after stage three and still capture most of the savings.

What Caching Does Not Fix

Caching does not reduce the cost of genuinely novel work. Workloads that are almost entirely novel, deep research, one-off document analysis, long agent trajectories with unique state, benefit little from any of the three; routing and prompt discipline remain the levers there.

It also does not fix a badly structured prompt. If the same information is resent because the application has no session state, the fix is application state rather than a cache that makes the waste cheaper.

And caching trades against personalization directly: the more a response is tailored to one user, the less any cache can be shared. That trade should be made deliberately at the product level rather than discovered when the hit rate fails to materialize.

Frequently Asked Questions

What is the difference between prefix caching and semantic caching?

Prefix caching is a provider-side optimization that skips recomputation over the stable leading portion of a prompt, saving input token cost while the model still generates the answer fresh. Semantic caching is an infrastructure-side lookup that returns a previously generated response for a request judged similar enough, skipping generation entirely. Prefix caching carries no correctness risk; semantic caching can return a wrong answer, which is why it needs an evaluation harness and prefix caching does not.

Why is my exact-match cache hit rate so low?

Because free-text human input is almost never byte-identical, and conversational requests carry accumulated history that makes each one unique by construction. Low single-digit hit rates are normal for that traffic shape. Exact matching earns its place on machine-generated or enumerable requests instead: pipeline stages hitting the same document, batch jobs with overlapping inputs, and evaluation runs replaying a fixed suite. If the traffic is natural language, semantic caching is the technique that produces real hit rates.

How do I choose a similarity threshold for semantic caching?

Empirically, never by default value. Sample real pairs of production requests, and at each candidate threshold have a judge or a human decide whether the stored response would have been acceptable for the new request. That produces a measured error rate per threshold, which lets the choice be made against the actual cost of a wrong answer. Then exclude categories that no threshold handles safely, particularly requests turning on negation, specific identifiers, amounts, or dates.

What should I put in the cache key?

Everything that changes the answer: the full prompt, the model identifier and version, sampling parameters such as temperature, the system prompt, tool or function schemas, any retrieved context injected into the request, and a tenant or user scope. Include the prompt-template version and the retrieval corpus version as well, so that editing a template or reindexing the corpus invalidates affected entries automatically rather than depending on someone remembering to flush.

Should caching sit in front of routing or behind it?

In front. A cache hit means no model is called at all, so the routing decision becomes irrelevant, and running the router first spends classification work on every hit for nothing. Both belong at the gateway, which is the layer that already sees every request and can enforce one key scheme, one invalidation policy, and one aggregate view of what the caching program is saving.