What is model routing? LLM routers explained
Model routing is the practice of choosing, for each request, which large language model (LLM) should handle it. A model router, also called an LLM router, sits in front of a pool of models, reads each request, and sends it to the model best suited to the job. Easy requests stop paying for a frontier model, and hard requests stop getting stuck on a model that is weak at them.
┌──────────────────────────┐
request ───▶│ model router │
└─────────────┬────────────┘
│ one model per request
┌─────────────────┼─────────────────┐
▼ ▼ ▼
small / fast mid-size model frontier model
model
How does model routing work?
Every routed request goes through the same three steps:
- Assess. The router reads the request and estimates which models in the pool can do the task.
- Filter. It keeps the models that are likely to succeed, and drops the ones that are not.
- Choose. It picks one of the remaining models, usually the cheapest, and sends the request there.
On Emissary, the router gives each model a confidence: the probability that the model completes this request. Models at or above your confidence threshold (0.80 by default) are the confident pool, and the cheapest model in that pool serves the request. If no model is confident, the request goes to your default model. Capability comes first and cost second, so savings never come from accepting worse answers. See how a routing decision is made for a worked example.
Why use model routing?
Most teams pick one model and send everything to it. Because that one model has to handle any request, they pick a strong, expensive one. Easy requests overpay, and hard requests are limited by whatever that single model happens to be bad at. Routing replaces the single choice with a decision per request:
- Better results than any single model. Every model is better at some tasks and worse at others, and those strengths don't line up on one easy-to-hard scale.
- Lower cost. A cost reduction of 30–50% is typical, and about 60% of requests are usually served by a model other than the default. These are typical figures, not guarantees. See how to reduce LLM costs with model routing.
- Lower latency. Smaller models respond sooner, and for agents that speed-up applies to every step that doesn't need a large model.
- Easier model migration. Adding or retiring a model doesn't mean rewriting application code. See migrate to new models.
- Insight into your workload. Every decision records each model's confidence and the routing reason. See what emerges from routing.
The full argument is in why route.
What are the types of model routers?
Routers differ along a few dimensions. The ones that matter most:
| Dimension | One end | Other end |
|---|---|---|
| What it optimizes for | Complexity-based: guesses how hard the task is, then picks a cheap or an expensive model | Performance-based: asks which models can actually do the task |
| What it knows about the models | Blind: a small decision model trained on labeled queries | Model-aware: trained on rollouts of each model, with one head per model |
| State | Stateless: each decision is made in isolation | Stateful: uses cache state, so long conversations and agent loops don't throw away paid-for context |
| Modality | Modality-blind: treats every request as text | Modality-aware: knows which models accept images |
| Cost | Per-token price only | Intelligence-density aware: prices in how many output tokens each model will need |
Emissary's router is performance-based, model-aware, stateful, modality-aware and intelligence-density aware. See the router design space for what each of these means, and what the router sees for the signals it uses.
What is the difference between a model router and an LLM gateway?
A gateway is infrastructure for how requests reach models: one interface over many providers, credentials, retries, rate limits, load balancing and failover. A router decides which model should handle each request. Many products do only one of the two. Emissary is a router that also works as a gateway: one URL, compatible with the OpenAI and Anthropic APIs, with a router behind it. You can also use its routing decision alongside a gateway you already run. See routers and gateways.
When should you not use model routing?
Routing is online model selection, made at request time. If you can select the model offline, do. When the steps of your system are fixed and each step's input is narrow and predictable, evaluate candidate models for each step and pin one. Routing is for the open-ended parts: chat, agents, and anything where you can't list every kind of request ahead of time. See when to route, and when not to.
How do you get started with model routing?
- Create a router in the Emissary dashboard: choose your models, add your own provider keys, and set a default model and a confidence threshold.
- Connect your app or agent by changing two things: the base URL and the
modelstring. The gateway works with the OpenAI and Anthropic SDKs, curl, and coding agents such as Claude Code. - Read the routing decisions in Activity to see each model's confidence and why the request went where it did.
- Run a trial on your own traffic before moving production traffic.
How does Emissary compare with other routers?
See compare LLM routers for Emissary side by side with OpenRouter, LiteLLM, Not Diamond and Router.com, and the router selection checklist for questions to ask of any router.
Frequently asked questions
Does model routing reduce quality?
It shouldn't, and it can improve it. A performance-based router only sends a request to a model that it is confident can complete it, and ranks by cost only among those models. Models are strong at different things, so routing each request to the model that handles it best can outperform any single model on your workload as a whole.
How much can model routing save?
A cost reduction of 30–50% is typical. The bottom of that range is typical when routing within one provider's model family, and the top when routing across open-source and closed-source models. Your result depends on your traffic, catalog and default model, so measure it on your own requests.
Does a router add latency?
A routing decision takes a little time, around 100 ms, and requests that go to a faster model finish sooner. For multi-step agents, one deflection to a smaller model in a ten-step process can make up the difference. Measure end-to-end latency on your workload rather than the routing step alone.
Can I route across providers?
Yes. A router's catalog can span providers, for example Anthropic, OpenAI and OpenRouter, using a key you store for each. See choose your models.
More questions are answered in the model routing FAQ.