LLM Model Routing in Production: When to Send a Query to a Small Model

Model routing is the practice of deciding, per request, which model should answer it, instead of sending every request to one default model. In a mature production system, sixty to eighty percent of traffic can be served by a small or mid-tier model with a quality fallback to a frontier model, and that change is the highest-leverage inference cost lever a platform team has, larger than prompt compression, caching, or renegotiating a per-token rate. The arithmetic is not subtle: on published price lists as of mid-2026, frontier-tier output tokens run on the order of ten to seventy five USD per million while small-tier output tokens run well under one USD per million, so moving half a workload down a tier changes the bill by an order of magnitude rather than a percentage. The hard part is not the models, which have been good enough for the routine majority of enterprise traffic for over a year. The hard part is the router: the component that must decide, in single-digit milliseconds, without access to an answer it has not generated yet, whether this request is one of the easy ones.

Key Takeaways

• Routing beats every other inference cost lever because it changes the price tier rather than the token count. Compression and caching cut what a request consumes; routing changes what it costs per unit consumed, and the tier gap is ten to seventy five times, not percent.
• There are four routing architectures, not one: static rules by task class, learned routers, cascades with escalation, and semantic caching as routing's degenerate case. Most production systems should layer two or three, with static rules as the outer shell.
• Cost per resolved request is the only cost metric that matters. Cost per token and cost per call both reward a router that sends everything to the cheap model and quietly fails, because the retry, the escalation, and the human handoff are invisible to both.
• The dominant failure mode is silent quality decay on the cheap path, not an outage. A degrading router throws no errors; it returns slightly worse answers to the majority nobody samples, and the signal surfaces weeks later as churn or support volume.
• Thresholds drift. A router calibrated against one model generation is miscalibrated the moment either endpoint changes, and the endpoints underneath changed again during the week of August 17, the ordinary cadence of this market.

Why Routing Is the Highest-Leverage Lever

Every inference cost program arrives at the same four options: use fewer tokens, cache more aggressively, negotiate a better rate, or use a cheaper model. The first three are bounded. Compression on an already well-written prompt returns twenty to forty percent before quality moves. Caching returns whatever the natural repeat rate of the traffic happens to be. Rate negotiation returns what the vendor is willing to give, and as the analysis of who actually controls the AI supply chain sets out, buyer leverage in a capacity-constrained market is weaker than procurement teams expect.

Routing is different in kind because it moves the request across a price boundary rather than shrinking it. A workload that shifts seventy percent of its traffic to a small model saves nearly all of that traffic's cost, not seventy percent of it, because the small-model price of the same volume is a rounding error against the frontier bill.

The counterargument is quality, and it deserves a serious answer. The honest claim is not that small models match frontier models; they do not, on hard reasoning, long-horizon planning, or unusual edge cases. The claim is that production traffic is not uniformly hard. Most enterprise applications skew heavily toward routine classification, extraction, summarization, retrieval-grounded answering, and short structured generation, with a genuinely difficult minority. The case for when smaller models suffice is made in the framework for small language models in the enterprise. Routing lets an organization act on that skew without betting the workload on the small model being good enough for everything, which it is not.

The Four Routing Architectures

Static Rules by Task Class

The router maps a request to a model using properties known before generation: the endpoint or feature it came from, the declared task type, input length, customer tier, tool-use requirements, or the tenant's data-residency constraint.

This is the least fashionable architecture and the one most teams should ship first. It is explainable, testable, deterministic under audit, adds effectively zero latency, and captures most of the available savings wherever the calling code already knows what work it is requesting. If the product has a "summarize this thread" button and a "draft a legal response" button, the routing decision was made in the product design and the router only has to read it.

The limit is real: static rules cannot see difficulty inside a task class. Every "answer the customer question" request looks identical to a rule that knows only the endpoint, so rules alone force the threshold to be set for the hardest member of the class, which routes the whole class up.

Learned Routers

A learned router is a small classifier, typically an embedding model with a lightweight head or a small model prompted to score, trained to predict which tier will handle a request acceptably.

Learned routers capture the within-class variance rules cannot. They are also the architecture most likely to consume a quarter and produce nothing usable. They need labeled training data, and the label for "would the small model have been good enough" requires running both models and judging both outputs across thousands of requests. They inherit a calibration problem, because a confidence score is useful only if it tracks actual quality. And they put a model in the request path, with its own latency, drift, and evaluation burden. The correct sequencing is to generate the labeled dataset as a byproduct of running a cascade in production, then train the router on it. Teams that train before they have production escalation data are guessing at the label distribution.

Cascade With Quality Check and Escalation

The cascade sends the request to the small model first, applies a quality check to the response, and escalates to the larger model only when the check fails. It matches how the decision should be made, because it decides after seeing an actual answer.

The quality check is the whole design. In rough order of cost and reliability: schema or structural validation, nearly free and catches malformed output; a self-reported confidence or refusal signal, cheap and weakly reliable; rules-based checks against known-bad patterns; a verifier model prompted to judge adequacy; and full model-as-judge evaluation. Most production cascades land on the verifier tier, because a small verifier judging a small generator still costs a fraction of one frontier call.

The cost profile is what teams model incorrectly. A cascade does not cost the small-model price. It costs the small-model price plus the check plus, on the escalated fraction, the full large-model price on top, because the discarded first attempt was already paid for. At a twenty percent escalation rate a cascade is decisively cheaper than routing everything up; at sixty percent it is slower and more expensive than not routing at all. That break-even is the number every cascade must monitor, and it moves when quality drifts. Escalated requests also pay a latency tax, waiting through the small model, the check, then the large model, which is why latency-sensitive paths put a learned router in front of the cascade rather than running a bare one.

Semantic Caching, Routing's Degenerate Case

Semantic caching answers a request from a stored response to a semantically similar earlier request, retrieved by embedding similarity rather than exact match. Treating it as routing rather than a separate optimization pays off three ways: the similarity threshold becomes a routing threshold subject to the same calibration discipline, cache hits get the same quality evaluation as any other route instead of being assumed correct because they were cheap, and staleness becomes a routing failure mode rather than a maintenance chore, which is the framing that gets it monitored. The retrieval mechanics underneath are the infrastructure covered in the decision framework for fine-tuning, RAG, and long context.

The Decision Matrix

The last row is the one most often omitted and the one most likely to cause an incident. Some traffic should be exempt by policy rather than by score, and that exemption belongs in a static rule at the outermost layer, because a learned router will eventually score a high-consequence request as easy: right about the difficulty, wrong about the consequence.

Building the Routing Decision

The task taxonomy comes first. A router cannot route without one, and most organizations discover they do not have one. The taxonomy is a finite list of task classes, each with a name, an owner, a volume share, a quality definition, and a consequence rating. Deriving it is instrumentation work: log a representative window of production requests, cluster them, and have the owner of each product surface confirm the clusters are real. It defines the golden sets, the escalation policies, the exemptions, and the reporting. Skipping it produces a router optimizing a global average across a traffic mix nobody has characterized, which is how a program saves forty percent of the bill while degrading the one class that mattered.

Confidence thresholds are set per class, from data, in configuration. Every non-static architecture reduces to a threshold: escalate above this difficulty score, accept below this verifier score, serve from cache above this similarity. Set them per class, because the acceptable escalation rate for internal document tagging and for customer-visible support responses are different numbers and one global value sets one of them wrong. Set them from a labeled distribution rather than intuition: run a few hundred requests per class through both tiers, judge both outputs, plot quality against score, and pick the point where the quality loss becomes unacceptable. Store them as versioned configuration deployable without a code release, because thresholds change more often than routing code does and one gated behind a deploy will not be adjusted when it needs to be.

Escalation should fire on more than the score. A production-grade policy escalates on verifier rejection, schema validation failure, an explicit refusal or low-confidence signal, a user-initiated retry, a safety-sensitive topic, and any tenant carrying a no-routing flag. The user retry is the most underused and most informative trigger, because a user pressing regenerate is a free, real-time label that the cheap path failed.

Evaluating a Router

Offline, golden sets per route. Each task class needs fifty to three hundred representative requests with an agreed quality judgment, versioned alongside the router configuration. They answer the question "would this change have degraded known traffic," which must be answerable before any threshold moves, so they belong in continuous integration, gating configuration changes the way unit tests gate code. They go stale silently as the traffic mix changes, so they need a refresh cadence and a named owner. The broader discipline is covered in the executive guide to AI evaluations; a router is one more system that cannot be operated without it.

Online, win rate between routes. Offline sets cannot capture live drift, so one to five percent of production traffic should be dual-routed: answered by the production route and also by the alternative, with the pair scored by a judge model or sampled human review. The resulting win rate, the share of paired comparisons where the cheap route is judged equal or better, is the router's health metric. A stable win rate means the routing decision is still valid; a declining one is the earliest signal of silent decay, typically weeks ahead of any user complaint.

Cost per resolved request is the metric that matters. Cost per token rewards a router that produces short bad answers; cost per call rewards one that fails cheaply and forces a retry. Neither is a business metric. Cost per resolved request is total inference spend attributable to a user need, divided by the number of user needs actually resolved. It counts the small-model attempt, the verifier call, the escalated frontier call, every user retry, and the amortized cost of downstream human handling, which makes it the only formulation under which a router that degrades quality looks worse rather than better on the dashboard. The denominator needs a resolution definition per class, another reason the taxonomy comes first: a session closed without human escalation, a record accepted downstream without correction. That definition will be imperfect, and an imperfect resolution metric still dominates a precise token metric because it points in the right direction.

Failure Modes

Silent quality decay on the cheap path. The defining risk. A router does not fail loudly; it returns adequate-looking answers that are slightly worse to the majority of traffic routed down, and because nobody samples the cheap path once it works, the decay is invisible until it surfaces as churn, support volume, or a retry rate nobody attributed to the router. The mitigation is structural: sample the cheap path continuously, forever, at a rate that does not decline once the project is declared done.

Threshold drift. Thresholds are calibrated against a specific pair of endpoints, and both update on the vendor's schedule rather than the buyer's. A small model that improves makes the threshold needlessly conservative, which is money left on the table; a frontier model that changes behavior makes the win-rate history non-comparable. Endpoint changes landed again during the week of August 17, which is why recalibration belongs on a schedule tied to endpoint version events rather than to quarterly planning.

Cache staleness. Semantic caching fails in the direction of confident wrongness: a fluent answer describing a state of the world that changed. Every cached class needs a time-to-live derived from how fast its underlying facts move, plus event-driven invalidation when the source data changes.

Escalation storms. When small-model quality drops or the verifier is misconfigured, escalation rate spikes, and a cascade escalating most of its traffic is both slower and more expensive than no routing. Escalation rate needs a monitored ceiling with a circuit breaker that pins the class to the frontier model and pages the owner.

Router latency and fail direction. A learned router adds a model call before the model call. Forty milliseconds to save cost on seventy percent of requests is a good trade; three hundred milliseconds on an interactive path is not. The router also needs defined failure behavior, and the correct default is fail-up: an unavailable router routes to the frontier model. Failing down is cheaper and wrong, because it converts a router outage into a silent quality incident.

Attribution loss. Once traffic splits across tiers, every downstream quality metric becomes a blend, and a blend cannot diagnose a tier-specific problem. Log route identity on every request and carry it into every quality dashboard from day one; retrofitting it after a question arises leaves the historical data unable to answer it.

A One-Quarter Deployment Sequence

The ordering is the point. Shadow mode before live routing converts routing from a bet into a measurement. Teams that skip instrumentation and taxonomy to start on the learned router finish the quarter with a classifier, no baseline to compare it against, and no way to prove it helped.

Build or buy runs alongside the sequence, since several vendors now ship routing as a gateway feature. The layer-by-layer defaults in the AI build versus buy framework apply cleanly: the routing mechanism is industry engineering and is reasonable to rent, while the taxonomy, the thresholds, and the golden sets encode the organization's own judgment about what a good answer is and should never be outsourced, whoever runs the gateway. The cost model underneath the whole exercise sits in the AI total cost of ownership framework.

Where Routing Does Not Help

Routing fails predictably in four cases. It does not pay below a few million requests a month, where engineering and evaluation cost exceeds the savings. It does not help when the task distribution is uniformly hard, because there is no easy majority to exploit. It compounds errors on multi-step agentic workflows, where one cheap step's mistake propagates through every step after it. And it should not ship in an organization unwilling to fund permanent quality sampling, because an unmonitored router is a quality incident with a delay fuse. A readiness test: if the team cannot name the metric that would tell them the router had started hurting users, and the person who looks at it, the program is not ready to go live.

Frequently Asked Questions

What percentage of production traffic can realistically be routed to a small model?

Sixty to eighty percent is the achievable range for typical enterprise application traffic, because most production workloads are dominated by routine classification, extraction, summarization, and retrieval-grounded answering rather than hard reasoning. The realistic number for a specific workload depends on its task distribution, which is why instrumentation and taxonomy come before any routing decision. Uniformly hard workloads, such as multi-step agentic planning, land far below that range.

Should a team build a learned router or start with static rules?

Start with static rules by task class. They capture most of the available savings wherever the calling code already knows the task type, they add no latency, and they are auditable. A learned router is the right second step, trained on labels generated by running a cascade in production, because that is where honest labels about which requests the small model could handle actually come from.

How is a cascade different from a router, and which is cheaper?

A router decides before generating; a cascade decides after generating a cheap answer and checking it. The cascade is more accurate about difficulty because it has seen an answer, but it pays for the small-model attempt plus the check plus the full large-model cost on every escalated request. It beats routing everything to a frontier model while the escalation rate stays low, roughly under a third, and becomes both slower and more expensive than no routing as that rate climbs.

What is the single most important metric for a production router?

Cost per resolved request, measured per task class. Cost per token and cost per call both improve when a router degrades quality, because neither counts retries, escalations, or unresolved user needs. Cost per resolved request counts the entire chain of attempts behind one user need, which makes it the only formulation where a router quietly hurting users looks worse rather than better on the dashboard.

How often do routing thresholds need recalibration?

Whenever either model endpoint changes version, which in practice means several times a year, on the vendor's schedule rather than the buyer's. An endpoint update invalidates a threshold in both directions: a stronger small model makes it needlessly conservative, and a changed frontier model makes the win-rate history non-comparable. Tie recalibration to endpoint version events and keep thresholds in versioned configuration so the adjustment does not require a code release.