AI Model Routing in 2026: Cut Agent Costs 60%

The practical 2026 guide to sending each agent request to the cheapest model that can still do the job, and stacking the discounts that get you past a 60% cut.

One company ran up a $500 million Claude bill after failing to set per-employee usage limits, and a single Priceline engineer's monthly token spend hit $40,000 - TechCrunch. Those are not typos. They are what happens when an AI agent, which re-reads its entire context on every single step, meets a flagship model priced at $25 to $50 per million output tokens. The bill grows quadratically while the value grows linearly, and by the time finance notices, the budget is gone. Uber exhausted its entire 2026 AI-coding budget by April, and Microsoft revoked some developers' coding licenses months after rollout, both in the same reporting.

Here is the structural problem: a static policy of "always use the best model" overpays for the majority of work that never needed the best model. Most of what an agent actually does (deciding which tool to call, pulling a field out of a document, classifying an intent, formatting a snippet of JSON) is easy. Easy work does not need frontier reasoning. It needs a model that is merely competent and consistent, and competent-and-consistent now costs 100 times less per input token than the frontier at the extreme end of the market. Model routing is the discipline of noticing that gap and acting on it: a cheap, fast classifier sits in front of a pool of models and sends each request to the smallest one that will still clear a quality bar.

This guide covers how routing actually works from first principles, the current 2026 price spectrum that makes it pay, the peer-reviewed evidence for what it really saves (and where the headline numbers are marketing), every serious platform from OpenRouter and LiteLLM to AWS Bedrock and Not Diamond, the stacked playbook that gets a real agent workload past a 60% cut, and the honest counter-narrative of where routing quietly backfires. It assumes no engineering background. It does assume you are tired of vendor slides that promise 90% savings without telling you the conditions.

Contents

1. Why agent costs explode (the mechanism, not the vibe)
2. The price spectrum: the arbitrage that makes routing pay
3. What model routing actually is
4. The evidence: what routing really saves
5. The platforms: gateways, routers, and clouds
6. How you actually get to 60%: the stacked playbook
7. Where routing backfires: the honest counter-narrative
8. A decision framework: when to route, when to pin
9. Implementation: wiring a router without breaking things
10. The future outlook: routing after the gateway wars

The master comparison: routing solutions scored

Before the detail, here is every serious routing and gateway option scored on the five things an operator actually cares about, on a 0 to 10 scale, ordered by weighted final score. The scores reward low overhead, smart routing, model breadth, control over your own data, and production durability. Read the justification in each cell, not just the number: a gateway that scores 6 on control because it inserts a third party in your request path is making a real trade you may happily accept.

The five criteria, and why each one carries the weight it does: cost overhead (25%) captures the fee a gateway adds on top of raw token prices, because a router that saves 40% while skimming 10% is a worse deal than it looks. Routing smarts (25%) rewards genuinely predictive selection over blunt rules, since the whole game is picking the right tier without a human. Model breadth (20%) matters because savings come from having a cheap floor to route down to, and a narrow catalog caps the arbitrage. Control and data (15%) rewards options that keep your prompts inside your own infrastructure, which is decisive for regulated data. Maturity (15%) penalizes anything in maintenance mode or run by a team that just pivoted, because a router is load-bearing infrastructure. Notice the top of the table is not a flashy marketplace but LiteLLM, an open-source proxy: when you weight for zero overhead and full control, self-hosting wins, and everything below it is a trade of control for convenience.

1. Why agent costs explode (the mechanism, not the vibe)

To cut a cost you first have to understand where it comes from, and agent costs come from a place that surprises almost everyone the first time they see the invoice. A chatbot is one request in and one response out, so its cost is roughly the length of the question plus the length of the answer. An agent is not one request. It is a loop: read the context, call a tool, read the tool's output, evaluate whether the goal is met, and iterate, often for dozens of steps on a single task. The load-bearing fact, the one that explains the entire cost structure, is that each step re-reads everything that came before it. The full context window gets resubmitted on every turn, so the token bill grows quadratically in the number of steps, not linearly. Doubling the number of steps more than doubles the cost. Cockroach Labs describes this as the exact reason production economics bear no resemblance to the numbers you saw in a pilot - Cockroach Labs.

The multiplier is measurable, not theoretical. A teardown by Spheron puts agentic workflows at 5x to 30x more tokens per task than a single chat completion, and it identifies the dominant driver clearly: input tokens, the re-read context, the tool schemas, the accumulating conversation history, not the output the agent produces - Spheron. This is the single most important insight in the whole cost-reduction argument, so it is worth stating plainly: agent spend is input-dominated, and that input is overwhelmingly repetitive. The same system prompt, the same tool definitions, the same retrieved documents, and the same growing transcript get shipped to the model again and again. Hold onto that fact, because it decides which cost lever matters most later, and it is the reason caching (which discounts repeated input) turns out to be a bigger lever than routing (which swaps the model).

The blowups are documented rather than hypothetical, which matters because this space is full of scare stories that do not survive scrutiny. The credible third-party anchor is TechCrunch's June 2026 investigation into runaway AI costs, which reported that per-developer token consumption rose roughly 18.6x in nine months on Jellyfish's data, and that Goldman Sachs projects global token usage multiplying 24x by 2030 - TechCrunch. The sharpest edge in that reporting is the retry loop. A task that fails partway and gets rerun with corrections can consume dramatically more tokens than a clean linear pass, and running the same task repeatedly can produce a 30x spread in consumption for identical work. OpenAI's Alexander Embiricos, quoted in the piece, captured the shift in customer conversations: they are "never about capability anymore," but about "spending visibility, auditability, token controls, and model efficiency."

The retry mechanism deserves a closer look, because it is where cost and quality stop being separable. When an agent takes a wrong step (a bad tool call, a malformed argument, a hallucinated file path) it rarely fails cleanly. It notices the error a step or two later and loops back to correct it, and every step of that correction re-reads the now-larger context. A single early mistake can therefore spawn a chain of expensive recovery turns, which is exactly why the same task run five times can vary in cost by an order of magnitude for identical work. This is the precise reason cost control and quality control are the same problem in agent systems: the cheapest thing you can do is get each step right the first time, and the most expensive thing you can do is shave a few cents off a step that then triggers a cascade of retries. Hold that thought, because it is the strongest argument against routing carelessly.

Why does this matter for routing specifically? Because the quadratic, input-heavy, retry-prone shape of agent cost tells you exactly where the savings live. If most of the bill is repeated input, then discounting repeated input (caching) and reducing input volume (trimming) attack the biggest pool. If the retry loop is the most expensive failure mode, then any cost-cutting move that raises the error rate can cost more than it saves, which is the central risk of naive routing. And if a large fraction of the steps are easy, then paying frontier prices for all of them is the waste that routing exists to eliminate. We build the argument on this foundation because a cost-reduction plan that does not start from the cost structure is just a list of tactics.

How you apply this, before touching a single router, is to get spend visibility per step. You cannot route what you cannot see. The first move for any team over a few hundred dollars a month in tokens is to instrument the agent so that every step's model, input tokens, output tokens, and cost are logged and attributable to a task. Our deeper breakdown of where the money actually goes in agentic systems is in The Cost of AI Agents, and the underlying economics of why intelligence got cheap enough to meter this finely are in The True Cost of LLM Inference in 2026. Once you can see the per-step cost, the routing opportunity stops being abstract: you will find, almost always, that a small number of hard steps dominate quality while a large number of cheap steps dominate the bill.

2. The price spectrum: the arbitrage that makes routing pay

Routing is not a clever trick or a discount you negotiate. It is arbitrage on a price spectrum, and if that spectrum did not exist, routing would save nothing. So the honest place to start is the spectrum itself, priced from the official pages as of July 2026. The Claude lineup runs from Claude Fable 5 at $10 input and $50 output per million tokens at the premium end, through Claude Opus 5 and Opus 4.8 at $5 and $25, Claude Sonnet 5 at an introductory $2 and $10 through the end of August 2026, down to Claude Haiku 4.5 at $1 and $5 - Anthropic pricing. Inside a single vendor's own catalog, the flagship costs five times the cheap tier on both input and output. That 5x is the smallest arbitrage on the board.

The other labs stretch the spectrum further. OpenAI's current family runs from GPT-5.6 Sol at $5 and $30, through GPT-5.6 Terra at $2 and $12 and GPT-5.6 Luna at $0.20 and $1.20, down to gpt-5-nano at $0.05 input and $0.40 output - OpenAI pricing. Google's Gemini line spans Gemini 3.1 Pro at $2 and $12 (rising to $4 and $18 above 200,000 input tokens), Gemini 3.6 Flash at $1.50 and $7.50, and Gemini 3.1 Flash-Lite at $0.25 and $1.50 - Google pricing. And the floor drops out entirely once you include open-weight and Chinese-lab models: DeepSeek V4 Flash at $0.14 and $0.28, with cache-hit input at $0.03, sits at the bottom of the frontier-capable range - DeepSeek. Kimi K2.6, GLM-5.2, and Qwen3 Max fill the space between, most of them under $1.50 input. We track the open-weight surge that created this floor in Top 10 Open Source LLMs in 2026.

The number that justifies the entire architecture is the ratio, not the individual prices. On output tokens, Claude Fable 5 at $50 against DeepSeek V4 Flash at $0.28 is a 179x gap. Claude Opus 5 at $25 against the same floor is 89x. GPT-5.6 Sol at $30 against gpt-5-nano at $0.40 is 75x, all inside one vendor. On input tokens, where agents spend most of their money, gpt-5-nano at $0.05 against GPT-5.6 Sol at $5.00 is a clean 100x. These are not percentages. They are multiples. And that distinction is the whole reason a headline like "cut agent costs 60%" is conservative rather than aggressive: if even half your steps drop from a flagship to a mid or cheap tier, your blended cost does not fall by a few percent, it collapses, because the tiers are separated by factors of five, ten, or a hundred.

Why does this spectrum exist at all, from first principles? Because a frontier model's price reflects its worst-case reasoning capability, the hardest thing it can do, while the distribution of real agent traffic is dominated by its easiest tasks. The labs deliberately segment capability into priced tiers because it is more profitable to sell competence cheaply and genius expensively than to charge one blended rate. That segmentation is a gift to anyone willing to route, because it means you can buy exactly the capability each step needs instead of buying genius for a formatting job. The mechanism is convex: cost rises steeply with the difficulty a model can handle, but most steps sit at the flat, easy bottom of the difficulty curve, and routing simply refuses to pay the steep price for the flat work. Our side-by-side of where the flagship tiers actually differ enough to matter is in Claude Opus 5 vs 4.8, and the full current benchmark-and-price grid lives in AI Model Benchmarks and Pricing.

Two discounts stack on top of whichever tier you route to, and a serious operator models all three together rather than treating routing as the only lever. Prompt caching discounts repeated input at roughly 90% across Anthropic, OpenAI, Google, and DeepSeek, which matters enormously for agents with a fat, stable system prompt. Batch processing takes a flat 50% off any work that tolerates a delay, on all major providers. The takeaway for the rest of this guide is that the spectrum runs from about $0.05 per million input tokens to $10, and from $0.28 per million output tokens to $50, and that 70x-to-179x gap is not noise. It is the raw material. Everything that follows is about harvesting it without breaking your agent.

3. What model routing actually is

Now that the economic case is clear, the mechanics are simple to state and easy to get wrong. A router is a small, fast decision-maker that sits in front of a pool of models and, for each incoming request, picks which model should handle it. The one iron rule is that the router must be far cheaper than the models it routes between, or the arbitrage collapses. If your routing decision costs as much as a model call, you have added a tax instead of a discount. In practice the good routers cost fractions of a cent per decision and add well under a fifth of a second of latency, which is why they are worth having.

Routing methods fall along a spectrum of sophistication, and the right choice depends on how much you trust an algorithm to judge difficulty. Rule-based routing is deterministic: send anything tagged "classification" to the cheap model, anything from the "planning" path to the flagship, route by user tier or request size. It adds zero machine-learning risk and near-zero latency, and it is where most teams should start. Semantic routing embeds the incoming prompt into a vector and matches it by similarity to labeled example prompts, choosing a route with no model call in the decision path at all. Cascade routing flips the order: it calls the cheap model first, checks the answer's confidence, and escalates to a stronger model only when the cheap one is unsure. Predictive routing is the most advanced: a trained meta-model predicts, before any inference, which candidate model will produce the best answer at the best cost, and picks accordingly.

The more predictive the method, the higher the potential savings and the higher the risk of a mis-route, sending a genuinely hard prompt to a model that quietly fails it. This is the central tension of the whole field. A rule that says "classification goes to the cheap model" cannot mis-route a hard reasoning task, because you never told it to route reasoning at all. A predictive router that decides every request is more powerful and more dangerous, because its mistakes are invisible until an eval catches them or a user complains. Understanding where a given product sits on this spectrum tells you exactly how much oversight it needs. A good mental model is that rule-based routing is a policy you wrote and can audit, while predictive routing is a judgment you delegated and must monitor.

It helps to see routing as a live product rather than an abstraction, and the clearest consumer-grade example is OpenRouter's Auto router, which analyzes each prompt and selects from a curated pool in under 60 milliseconds, less time than it takes to stream a single output token, at no fee beyond the chosen model's rate - OpenRouter. The following overview walks through how gateway-based routing across many providers looks in practice, which is useful before we get into the specific platforms.

Why this matters and how to apply it: the method you pick should match the maturity of your monitoring, not the sophistication you can brag about. If you cannot yet run a nightly eval to catch silent quality regressions, start with rule-based routing on the tasks you are certain are easy (extraction, classification, formatting) and leave everything else on a single strong model. Graduate to semantic or predictive routing only once you have the eval loop to catch its mistakes. Routing is one layer in the larger discipline of designing good agent loops, which we cover in How to Write Loops for AI Coding Agents, and the model-selection decision is a natural extension of the orchestration patterns in Multi-agent orchestration.

4. The evidence: what routing really saves

Every routing vendor has a savings number on its landing page, and almost all of them are workload-specific best cases dressed up as laws. To reason honestly, you have to separate the peer-reviewed evidence from the marketing, and the good news is that the research is strong enough that you do not need the marketing. The foundational result is RouteLLM, from LMSYS and UC Berkeley, published in 2024 and later accepted at ICLR 2025. It trained routers on human preference data to decide, per query, whether a request needed a frontier model or could be handled by a small open one, and reported cost reductions of over 85% on the MT-Bench chat benchmark, 45% on MMLU, and 35% on GSM8K, while retaining 95% of the frontier model's quality - LMSYS.

Read that spread carefully, because it is the most important lesson in this section. The savings are not one number, they are a range from 35% to 85%, and the range is entirely explained by how easy the workload is. On open-ended chat, the router hit its quality bar while sending only 14% to 26% of calls to the expensive model, because most chat turns are easy. On a dense knowledge exam like MMLU, the same framework needed the strong model on 54% of calls, because most exam questions are hard. Routing pays in direct proportion to how much of your traffic is genuinely easy. Agent glue-work (tool selection, extraction, formatting) skews easy, which is good news for anyone reading this guide, but exam-style reasoning skews hard, and no router will save you 85% on a workload that is mostly hard. The image below is the single most quotable chart in the field: the trained router's quality-per-dollar curve sits well above the random-routing baseline at every price point.

Cascades are the older, complementary idea, and they come with their own famous number that deserves the same skepticism. FrugalGPT, from Stanford in 2023, showed that a cascade (call the cheap model, escalate only when a scorer flags a weak answer) could match a frontier model's performance with up to 98% cost reduction on certain question-answering datasets - arXiv. The 98% is real but it is a best case on tasks where a cheap scorer can reliably tell when the cheap model was already right. Treat it as "cascades can, in favorable conditions, remove almost all frontier calls," not "you will save 98%." More sober and more portable are two later results: AutoMix (NeurIPS 2024) reported over 50% cost reduction for comparable performance using self-verification to decide when to escalate - Hugging Face, and Microsoft's BEST-Route (ICML 2025) reported up to 60% cost cut with less than 1% performance drop by jointly routing the model choice and the number of sampled responses - Microsoft. That BEST-Route figure is the single strongest research-grade citation for a 60% number at near-zero quality loss, which is precisely the target this guide is named after.

The most conservative and therefore most useful number comes from a hyperscaler with no incentive to overpromise. AWS states that Amazon Bedrock Intelligent Prompt Routing can "reduce costs by up to 30 percent without compromising on accuracy," and its own internal testing broke that down by model family: 35% average savings on the Nova family, 56% on the Anthropic family, and 16% on the Meta family, with routing overhead of about 85 milliseconds - AWS. Note the honesty of a 16% figure sitting next to a 56% one: same technology, different workloads, wildly different outcomes. There is a "cut costs by 94%" Bedrock number circulating on third-party blogs, and it appears nowhere in AWS's official materials, which top out at 56% under favorable conditions. That gap between the marketing echo and the vendor's own floor is the whole reason this section exists.

Why this matters and how to apply it: the defensible planning band for routing alone is 30% to 60% on the requests you actually route, which is a minority of traffic, not the whole bill. Commercial routers report even more modest figures when they are honest. Not Diamond, the predictive router that powers OpenRouter's Auto mode, reports roughly 5% accuracy gains and about 30% cost savings, framed as vendor-reported and directional - Not Diamond. So the rule for setting expectations with your own finance team is: promise 30% from routing, treat anything above 50% as workload luck, and get the rest of the way to 60% from the stacked levers we cover in section 6. If you want the exhaustive current picture of which evals still predict real-world agent performance, that is Top 50 AI Model Evals.

5. The platforms: gateways, routers, and clouds

The market has organized itself into four layers, and understanding which layer a product lives in tells you more than any feature list. There is the marketplace layer (one API to reach every model), the managed gateway layer (observability, governance, and routing as a service), the pure routing-research layer (a smart selection engine that sits on top of everything else), and the cloud-native layer (routing as a checkbox inside AWS, Azure, or Databricks). No single product is best; the right one depends on whether you value control, convenience, or already living inside a hyperscaler. We map the adjacent world of tool and integration gateways in LLM Tool Gateways, which pairs naturally with model routing.

The breakout of the marketplace layer is OpenRouter, a single OpenAI-compatible endpoint fronting more than 400 models from over 60 labs. In May 2026 it more than doubled its valuation to $1.3 billion on a $113M Series B led by CapitalG, and it now processes roughly 100 trillion tokens per month across 8 million users - TechCrunch. Its economics are the important part: OpenRouter passes provider token rates through at cost and monetizes on a flat 5.5% fee on credit purchases, having dropped its older per-token markup, and its Auto router adds no fee beyond the selected model's rate. The moat is not the router. It is being the default place every new model appears on launch day, which makes OpenRouter the fastest way to try routing across labs without signing up for each one.

The managed gateway layer is anchored by Portkey and LiteLLM, which represent the two ends of the build-versus-buy choice. Portkey is a full control panel fronting 1,600+ LLMs with the widest routing menu of any managed player: conditional routing on metadata or request parameters, load balancing, fallback, and semantic routing, all expressed as configuration rather than code, on a log-based plan starting at $49 per month. Two 2026 events tell you where this layer is heading: Portkey open-sourced its gateway in March and was acquired by Palo Alto Networks in May, folding into a security platform - TrueFoundry. LiteLLM, by contrast, is the open-source workhorse: a self-hostable proxy that normalizes 140+ providers and roughly 1,892 models into one format, free to run forever, where you pay only your infrastructure and your tokens - LiteLLM. LiteLLM tops our scored table precisely because it adds zero toll and keeps every prompt inside your own walls.

The pure routing layer is best represented by Not Diamond, which does one thing extremely well: it predicts the best model per query from evaluation data in under 60 milliseconds and is stack-agnostic, sitting on top of gateways rather than replacing them, which is why OpenRouter uses it for Auto routing - VentureBeat. Its funding tells a story worth heeding: a modest $2.3M seed but with angels including Google DeepMind's Jeff Dean and Databricks' Ion Stoica. The cautionary counterexample in this layer is Martian, an early standalone router that raised a $9M seed from NEA and General Catalyst in 2023 and took an Accenture Ventures investment in 2024, but has since pivoted toward AI interpretability research - The SaaS News. The lesson is structural: a standalone router is a hard business to sustain when gateways bundle routing for free, which is why the durable routing companies either become gateways or embed inside something larger.

The cloud-native layer turned routing into a one-line configuration for anyone already inside a hyperscaler. AWS Bedrock Intelligent Prompt Routing offers a single serverless endpoint that routes within one model family (a smaller and larger Claude tier, or a smaller and larger Llama) by predicting response quality per request, at no cost beyond the chosen model - AWS. Azure AI Foundry's model router does real-time cross-model selection with quality, cost, and balanced modes across a pool that now spans several vendors' current models, and Databricks Mosaic AI Gateway offers the same control plane for teams that live in the lakehouse. The diagram below shows the Bedrock mechanism: requests are directed between two models in the same family based on which one the router predicts can handle the specific prompt.

Two more managed options round out the practical shortlist, and both compete on being nearly free at the gateway itself. Cloudflare AI Gateway keeps its core features (analytics, caching, rate limiting) free and charges an optional 5% billing fee, adding dynamic routing and data-loss-prevention at the edge - Cloudflare. Vercel AI Gateway takes zero markup on tokens and includes $5 of monthly credits, positioning routing as a resilience feature (it retries the identical model on a backup provider when one degrades) rather than a cost optimizer - Vercel. For an agent-workforce platform where routing lives one layer up, inside the orchestration itself, O-mega selects which model runs each sub-task across its internal, browser, and computer session types, so the routing decision is made per step by the platform rather than configured in a separate gateway. It belongs on this list as one option among many, with the same trade-off as any managed layer: less to operate, less control over the internals.

Between the marketplaces and the clouds sits a band of specialist gateways that compete on routing depth rather than raw reach. Requesty unifies 400 to 600+ models with sub-14-millisecond failover and markets aggressive savings, though its headline "up to 80%" figure is self-reported rather than audited, so treat the mechanism as sound and the magnitude as workload-dependent - Requesty. Kong AI Gateway brings API-management pedigree, adding semantic routing by a configurable similarity threshold and six load-balancing algorithms across the major providers, which is the natural choice for a team that already runs Kong for its non-AI traffic - Kong. Unify takes the most research-forward stance, benchmarking every provider endpoint continuously and routing through a neural scoring function with sliders for quality, cost, and latency. The through-line across this band is that routing intelligence, not model count, is the differentiator, and it is where the predictive methods from section 3 show up as shipped product features rather than papers.

Why this matters and how to choose: the layer decides the trade, and the trade is always control versus convenience. If your data is sensitive and your volume is high, self-host LiteLLM (or the open-source RouteLLM framework) and keep every prompt in your infrastructure. If you want speed-to-market and resilience over cost-squeezing, a managed gateway like Portkey or Cloudflare earns its fee. If you already live inside AWS or Azure, the native router is free and good enough for intra-family savings without adding a vendor. The 2026 consolidation wave (Portkey into Palo Alto Networks, Helicone into Mintlify, Martian pivoting away) tells the deeper story that standalone routing is becoming a commodity feature absorbed into larger platforms, so weight durability heavily and avoid betting your stack on a single-feature startup.

6. How you actually get to 60%: the stacked playbook

Here is the claim that undoes most of the marketing: the 60% is not a routing number. It is a stack. Routing alone, as section 4 established, saves 30% to 60% on the minority of requests you can safely route down, which contributes a modest slice of the whole bill. The way real teams reach 60% overall is by combining routing with three orthogonal levers that attack different pools of cost, and because they apply to different pools, they compound rather than simply adding up. The structural anchor is the fact we established in section 1: agent cost is input-heavy and repetitive, which means the biggest lever is not the model swap at all.

The biggest lever is prompt caching, and it is the one most teams under-use. Because an agent resends the same large prefix (system prompt, tool schemas, retrieved documents, accumulating history) on every turn, discounting that repeated input attacks the largest cost pool directly. Anthropic charges cache reads at 0.1x the base input rate, a 90% discount, after a one-time write premium of 1.25x, and AWS quotes the same order of magnitude for Bedrock - Anthropic. Net of imperfect hit rates and the write premium, budget caching to remove 30% to 40% of the total bill. The break-even is trivial: a 5-minute cache write costs only a 1.25x premium and every read after it costs 0.1x, so you recover the premium after a single hit and everything afterward is pure saving.

The catch that most teams learn the hard way is that caching only pays if your prefix is genuinely stable and your requests arrive inside the cache window. Anthropic's default cache lives for five minutes and can be extended to an hour at a higher write premium, so an agent that fires steps in quick succession sees near-total hit rates while one that idles between steps keeps paying to rewrite the same prefix. The engineering implication is concrete: put the truly static content (system prompt, tool definitions, policy text) at the very front of the context so it forms a stable cacheable block, and keep the volatile content (the latest user turn, fresh tool output) at the end where a cache miss costs the least. Teams that structure their prompts this way routinely see the majority of their input tokens billed at the 0.1x read rate, while teams that interleave static and dynamic content get a fraction of the benefit for the same nominal feature.

The second lever is context trimming, and it is the highest-return engineering work after caching because it carries no quality-regression risk when done well. Trimming reduces token volume at the source: prune verbose tool outputs before they enter the context, cap how much history the agent carries, and summarize old turns instead of resending them verbatim. This attacks the quadratic base that everything else multiplies against, so a 15% reduction in average context length compounds across every step of every task. Budget it at 10% to 15% of the bill. Only after caching and trimming does model routing enter, contributing its 10% to 15% by downshifting the safely-easy subset of steps, and finally the batch API takes a flat 50% off any non-interactive work (overnight evals, bulk enrichment, offline summarization) for another 3% to 5%.

Summed conservatively, those layers reach roughly 63%, with wide variance depending on your workload. The honest decomposition is that caching and trimming get you to around 45% to 50% with no quality risk, and routing and batching add the rest while introducing the trade-offs we cover next. Anyone selling "60% from routing alone" has inverted the actual cost structure, because they are ignoring that the dominant cost in an agent is repeated input, not the model. The before-and-after below shows the same agent step with all four levers applied.

A fully worked example makes the arithmetic concrete. Take an agent that runs 100 sub-steps, each with about 2,000 input and 500 output tokens, priced on Claude Opus at $5 input and $25 output. Run every step on Opus and each costs $0.0225, for a total of $2.25. Now route the 78 easy steps to Haiku 4.5 at $1 and $5 (each Haiku step costs $0.0045) and keep the 22 hard steps on Opus. The routed total is $0.351 plus $0.495, or $0.846, a 62.4% saving from the price gap and the easy/hard split alone, before any caching. Layer caching on the stable 1,500-token prefix at 0.1x and the combined saving pushes past 70%. That is the entire thesis in one calculation: a 5x price ladder, times a heavy-tailed workload, times a cached prefix. The broader efficiency toolkit, including compression and quantization approaches that lower the unit price of every tier, is in How to Cut LLM Costs and Google TurboQuant.

7. Where routing backfires: the honest counter-narrative

A guide that only sells the upside is a brochure, and routing has real failure modes that can turn a cost-saving initiative into a net loss. The most fundamental is that a router is non-deterministic by default. The same prompt that hits model A today may hit model B tomorrow as pricing, availability, or the router's heuristics shift. That breaks the thing engineering teams rely on most: reproducibility. You can no longer say "the agent does X," because which model produced X is a moving target, which undermines regression testing, auditing, and every debugging session that starts with "it worked yesterday" - OpenLegion. For a regulated or safety-critical agent, this alone can disqualify aggressive routing.

The second failure mode is the one that connects directly back to why agents are expensive: a silent quality regression compounds through the retry loop. In a single-turn chatbot, a slightly worse answer is just a slightly worse answer. In an agent loop, one bad decision from a downshifted model triggers a retry, and the retry is the single most expensive event in the entire system. Routing a step to save 30% can increase total cost if it raises the retry rate even modestly, because a retried task re-reads its whole context and runs again. This is why the counter-intuitive advice holds: the steps you must never route down are the ones early in a long chain, where a cheap mistake cascades into dozens of expensive corrections. The cheap model is safe on a leaf task and dangerous on a root decision.

There are four more traps that each deserve a sentence of respect. The latency tax is real: every gateway hop adds request-path time, and AWS alone quantifies its own routing overhead at about 85 milliseconds, before you add a third-party gateway's own embedding lookups and policy evaluation. Gateway lock-in is a genuine paradox, because the multi-vendor gateway you adopt to avoid provider lock-in becomes its own dependency as provider details leak into your prompts, tool schemas, and cost controls, and the exit becomes expensive - Medium. Semantic caching, the aggressive cousin of prompt caching that serves a stored answer when a new query is merely similar to an old one, can leak one user's response to another if the cache is scoped wrong, which is a privacy failure and not just a wrong answer - TrueFoundry. And routing your prompts through an external gateway means your proprietary context and any PII transit a third party, which is a direct compliance exposure for regulated data.

Routing also introduces an attack surface, which most cost guides omit entirely. Research on router robustness shows that preference-data routers are fragile across task categories and can be manipulated - arXiv, and a 2026 paper demonstrated a black-box adversarial-suffix attack that reliably forces routers to select the expensive model, inverting your savings on purpose. If an attacker can append a crafted string to inputs your agent processes, they can run up your bill. This is not a reason to avoid routing, but it is a reason to rate-limit, monitor per-user spend, and treat the router as security-relevant infrastructure rather than a passive cost optimization.

The most important counter-argument is also the simplest, and honesty requires stating it fully: you might not need a router at all. With a strong cheap model like Gemini 3.1 Flash-Lite or Haiku 4.5 delivering 80% to 90% of frontier quality on well-scoped tasks, many teams capture most of the savings by simply defaulting to one good cheap model and reserving the flagship for explicitly-flagged hard calls, with no router, no gateway, and no non-determinism. The rebuttal is that mixed-tier routing still shows meaningfully better cost-efficiency and higher throughput than a single-model baseline on suitable workloads - FindSkill. So routing wins on efficiency, but the single-cheap-model default wins on simplicity, auditability, and debuggability. That is a real trade-off to weigh, not a free lunch, and for a low-volume or uniformly-hard agent the simple default is often the correct engineering call.

8. A decision framework: when to route, when to pin

The preceding sections give you the pieces; this one assembles them into a decision you can actually make. The first question is not "which router" but "should I route this task at all," and the answer comes from sorting your agent's steps into two buckets. Route down the tasks that are reflexive and well-scoped: intent classification, field extraction, tool-argument formatting, and short summaries, where a cheap model is not just adequate but sometimes better, because frontier models can overthink a simple decision and add latency. Keep on the flagship the tasks that are open-ended and consequential: long-horizon planning, hard multi-step reasoning, and ambiguous synthesis, where the coding and reasoning benchmarks still show a 7-to-10-point gap between the value tier and the frontier, and where that gap compounds across an agent's steps.

The second question is build versus buy, and it turns almost entirely on volume and data sensitivity. Below a few thousand dollars a month in tokens, the engineering cost of building, monitoring, and re-tuning a router usually exceeds the savings, so the right move is either a managed gateway you can turn on in an afternoon or the single-cheap-model default from the last section. Above that threshold, and especially with sensitive data, self-hosting an open-source proxy like LiteLLM (optionally with a predictive layer like Not Diamond on top) pays for itself and keeps your prompts in your own infrastructure. The crossover point is not a fixed dollar figure, it is where the monthly savings clear the loaded cost of the engineer-hours to maintain the router, which for most teams lands somewhere in the low thousands of dollars per month.

The third question is which layer, and the framework here is a short chain of conditionals rather than a menu. If you already run inside AWS, Azure, or Databricks, start with the native router, because it is free, adds no vendor, and keeps data in your cloud, accepting that it mostly routes within one model family. If you are provider-agnostic and want the widest catalog with the least setup, a marketplace like OpenRouter gets you routing across labs in an afternoon at a 5.5% fee. If you need governance, observability, and fine-grained routing policy, a managed gateway like Portkey is built for exactly that. And if data residency or cost control is paramount, self-host. Notice that this chain never produces "the best router," only "the right router for your constraint," which is the honest shape of the decision.

Why this matters and how to apply it: the biggest mistake teams make is starting with the router instead of the eval. Before you route a single request, build the quality gate: a small, representative test set that runs nightly and flags when a routed step's output quality drops below a threshold. Routing without an eval is not a cost optimization, it is a silent quality gamble that surfaces as retries and user complaints weeks later. Once the eval exists, start conservative (rule-based routing on the tasks you are certain are easy), measure the savings and the quality delta, and expand the routed set only as the data justifies it. This is the same discipline that governs any production agent, which we treat end-to-end in Building AI Agents.

9. Implementation: wiring a router without breaking things

Concreteness matters here, because the gap between "routing sounds great" and a working router is mostly plumbing. The fastest path for a provider-agnostic team is a marketplace endpoint. With OpenRouter, you point your existing OpenAI-compatible client at a single base URL and change the model string, and the Auto router handles selection with no other code change.

A minimal Auto-routing call looks like this:

curl https://openrouter.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openrouter/auto",
    "messages": [{"role": "user", "content": "Extract the invoice total."}]
  }'

The self-hosted path gives you control at the cost of running the proxy, and the reference implementation is the open-source RouteLLM framework, which is deliberately an OpenAI-compatible drop-in: you configure a strong model and a weak model, choose a router type, and set a threshold that decides how much traffic goes to the strong model. The architecture diagram below shows the shape: your client talks to the router exactly as it would talk to a model, and the router decides internally which backend to call.

Running RouteLLM as a local drop-in server takes a few lines:

pip install "routellm [serve,eval]"
python -m routellm.openai_server \
  --routers mf \
  --strong-model "your-flagship-model" \
  --weak-model "your-cheap-model"
# then point any OpenAI client at http://localhost:6060/v1

For teams that want rule-based or semantic routing rather than a trained model, the two workhorses are LiteLLM (a config-driven proxy that maps model names to providers with fallback and load balancing) and the open-source semantic-router library, which routes by embedding similarity with no model call in the decision path - GitHub. The pattern is the same across all of them: define your routes, set your cheap and expensive backends, and put the whole thing behind the OpenAI-compatible interface your application already speaks. The important engineering discipline is that the routing policy lives in one place, versioned and reviewable, rather than scattered across your application code as ad-hoc model-name strings.

The step almost everyone skips, and the one that separates a real cost win from a quiet quality loss, is the evaluation harness. A router without continuous evaluation is a fit-and-forget system in a world where models change monthly and your traffic drifts weekly. The minimum viable eval is a set of a few dozen representative tasks with known-good outputs, scored nightly, that alerts when a routed step's quality falls below a threshold. When a new cheap model launches (and in 2026 that happens constantly), you re-run the eval before expanding the routed set to include it, rather than trusting the leaderboard. This is exactly the self-improving-loop discipline we describe in Self-Improving AI Agents: the router improves only if something measures whether its decisions were right.

Observability is the other half of the eval, and it is cheaper to build than most teams assume. Every serious gateway logs, per request, which model was selected, the input and output token counts, the cost, and the latency, and the single most valuable dashboard you can build from that data is a per-task cost breakdown that shows which steps consumed the budget. The pattern that emerges is almost always the same: a small number of hard steps dominate quality while a large number of cheap steps dominate the invoice, and until you can see that split you are routing blind. Pair the cost dashboard with a routing-decision log (what the router chose and why) so that when a quality regression appears in the nightly eval, you can trace it to the specific mis-route that caused it rather than guessing. Cost visibility and quality visibility together turn routing from a leap of faith into a controlled experiment.

Why this matters and how to apply it: sequence the rollout so that risk is always bounded. Start by putting a gateway or proxy in front of your existing single-model setup with routing turned off, so you gain observability and per-step cost data without changing behavior. Then enable rule-based routing on one clearly-easy task category and watch the eval for a week. Then, and only then, expand to more categories or a predictive router. At every stage you should be able to answer "how much did we save and did quality drop," and if you cannot, you have moved too fast. The teams that get burned are the ones that flip on aggressive predictive routing across all traffic on day one and discover the retry-driven cost spike a month later.

10. The future outlook: routing after the gateway wars

Zoom out and the direction of travel is clear from the 2026 consolidation. Standalone routing is becoming a commodity feature, not a durable standalone product. Portkey was absorbed into a security platform, Helicone into a documentation company, and Martian pivoted away from routing entirely, all inside a few months. The lesson is not that routing is unimportant, it is that routing is too useful and too easy to bundle to survive as a business by itself. It is migrating into three homes: the gateways that already front your models, the security platforms that already inspect your traffic, and the cloud consoles you already pay. For an operator, this is good news, because it means routing is trending toward free and built-in rather than a line item.

The second structural force is the rise of the cheap floor, which quietly widens how much traffic a router can safely divert. OpenRouter's own usage data shows open-weight models now accounting for roughly a third of all token volume, with DeepSeek alone at over 14 trillion tokens in a year - Crypto Briefing. This is the "why now" for routing: the cheap end of the menu got dramatically more capable, so the quality penalty for routing a query down-market shrank, which mechanically increases the fraction of your workload that routes safely. Every time a new open-weight model closes the gap on the frontier for a task class, the addressable savings from routing that class go up without you changing anything. The platform's explosive growth also shows how much volume is now in play.

The third force is that routing is moving up the stack, from a gateway concern to an orchestration concern. In a single-model chatbot, routing is an infrastructure detail. In a multi-agent system where a lead agent spawns ephemeral sub-agents for browser work, computer work, and internal reasoning, the model-selection decision is inseparable from the orchestration itself: the system that decides what sub-task to spawn is the natural place to decide which model runs it. This is why agent-workforce platforms like O-mega treat model selection as a per-sub-task decision made by the orchestrator rather than a gateway configured underneath it, and it is where the most sophisticated routing will increasingly live, because the orchestrator has context the gateway does not (it knows whether this step is a root planning decision or a leaf extraction). The economics of that shift, where cheap intelligence gets multiplied into outcomes by the systems that apply it, are the subject of The Big Pipe and The agent economy.

Reasoning from first principles about where this ends: as long as the labs keep segmenting capability into priced tiers, and as long as agent workloads keep being heavy-tailed in difficulty, the arbitrage that routing exploits will persist. The mechanism does not depend on any specific model or vendor, so it survives every model release. What changes is the magnitude and the tooling: the cheap floor rises, closing the quality gap and widening the safe-to-route fraction, while the tooling gets absorbed into platforms and trends toward free. The durable skill is therefore not "learn OpenRouter" or "learn LiteLLM," it is the discipline this guide teaches: measure your per-step cost, understand your workload's difficulty distribution, match each step to the cheapest tier that clears the bar, and stack caching and batching on top. That skill compounds as the market moves, while any specific tool choice is a one-year bet. The swarm-architecture version of this discipline, where cost-efficiency comes from how you shape the agent fleet itself, is in Kimi K2.6 Agent Swarm.

Conclusion: the decision, in one page

Model routing works because the market gave you a 70x-to-179x price spread and a workload where most steps are easy, and routing is simply the discipline of matching each step to the cheapest tier that still clears a quality bar. The peer-reviewed evidence is solid (RouteLLM at 95% quality with only 14% to 26% of calls on the flagship, Microsoft's BEST-Route at 60% savings with under 1% quality loss), but the honest planning number for routing alone is 30% to 50% on the requests you can safely route, not the whole bill.

The path to a real 60% cut is a stack, not a single move. Caching and trimming get you to roughly 45% to 50% with no quality risk, because agent cost is dominated by repeated input, and routing plus batching add the rest while introducing non-determinism and the retry risk you must monitor. So the decision framework is: instrument per-step cost first, build a nightly eval second, turn on caching and trimming before anything else, add rule-based routing on your provably-easy tasks third, and reach for a predictive router or a managed gateway only once the eval can catch its mistakes.

On tooling, let the constraint pick the layer. Self-host LiteLLM or the RouteLLM framework when data and control matter, use a marketplace like OpenRouter when breadth and speed matter, lean on your cloud's native router when you already live in AWS or Azure, and consider an agent-orchestration platform like O-mega when routing is one decision inside a larger autonomous workflow rather than a standalone concern. Whatever you choose, remember the trade the whole field runs on: routing buys you efficiency, and it costs you determinism, so the cheapest agent is not the one on the cheapest model, it is the one where every step runs on exactly the model it needed and nothing more.

This guide reflects the AI model routing landscape as of July 2026. Model names, prices, and platform ownership change constantly (three routing vendors were acquired or pivoted in the first half of 2026 alone), so verify current pricing and availability against the official pages before committing. Written by Yuma Heymans (@yumahey), founder and CEO of O-mega and co-founder of the recruitment-agent company HeroHunt.ai, who works on production AI-agent products between Amsterdam and San Francisco and spends more time than he would like on the per-query model-cost tradeoffs this guide describes.