The Thinnest Layer with the Most Leverage

Model routers, AI economics, and where value settles

Most of the money in enterprise AI is now being allocated by a component almost nobody in the C-suite has reviewed.

It is called a router. It sits between an application and a fleet of models, and it decides, prompt by prompt, which model gets the work. It is usually a few hundred lines of code and a small classifier. It is also, increasingly, the layer that determines which vendors capture value in this market and which ones become interchangeable.

This is a short explainer on how routers work, the tradeoff enterprises are managing when they deploy one, and what the whole thing means for open systems and for the frontier labs.

The routing market is built in the control plane, claimed from above and below by cloud, model, and developer-tier vendors, and governed almost nowhere.

I. What a Model Router Does

The premise is simple. Not every prompt needs a frontier model. "Reset my password" and "reconcile these three contradictory clauses across a 90-page agreement" are not the same computational problem, and until recently most enterprises paid the same price for both.

A router classifies the incoming request and dispatches it to the cheapest model likely to handle it correctly. There are four common designs, in ascending order of sophistication:

Rules and keywords. Regex, intent labels, string matching. Adds under a millisecond. Brittle, but transparent, and it handles the obvious cases.

Embedding similarity. The prompt is embedded and compared against clusters of known query types with known model requirements. Roughly five milliseconds.

Learned classifiers. A small trained model predicts whether the cheap model will produce an acceptable answer. Fifty to a hundred milliseconds, and materially better at the ambiguous middle.

Cascading. Answer with the small model first. Run a confidence or verification check. Escalate only on failure. This trades a little latency for the strongest cost profile, because the expensive model is called only after the cheap one has demonstrably failed.

Router overhead is negligible against inference. A typical model response takes 500 to 2,000 milliseconds. Even the heaviest classifier is a rounding error on that.

The savings are not marginal. RouteLLM, published at ICLR 2025, maintained 95% of GPT-4 Turbo's answer quality on MT-Bench while routing only 14% of queries to the strong model, for roughly 85% cost reduction. Ordinary production splits behave similarly. Send 80% of traffic to a cheap model and 20% to a frontier model and you land near a 79% reduction on input costs, with the spread widening on output-heavy workloads.

II. The Tradeoff, and How Enterprises Are Adopting It

Routing is not free money. You are buying a cost reduction with a probability of worse answers. The trade is almost always worth making. It is only safe to make if you can see what you gave up.

Two properties make that hard.

The failure mode is silence, not error. A misrouted prompt does not return a stack trace. It returns a plausible, confident, marginally degraded response that satisfies your monitoring, passes your uptime check, and surfaces four days later as a customer complaint, a bad recommendation, or a compliance finding. Nothing in a standard observability stack catches it.

The two sides of the trade land in different quarters and different P&Ls. The savings appear on next month's invoice, precisely measured, attributable to the team that delivered them. The quality cost appears months later as churn, escalations, or a deal that went quiet, attributed to something else entirely. One side of the ledger is legible and immediate. The other is diffuse and delayed. Every incentive in the organization is therefore pointed at cutting cost and away from noticing what it cost you.

That asymmetry, not the technology, is why routing programs go wrong.

If your organization has deployed a router without an evaluation gate, you have not implemented a cost optimization. You have implemented an undocumented quality reduction with a plausible business case attached.

How are routers being adopted in the enterprise?

Routing is entering enterprises through three doors, and it is worth knowing which one it came through in your company, because each carries a different blind spot.

Developer-led. A team adds a gateway, sets defaults, and ships. No procurement event, no security review, no sign-off. This is the most common path by a wide margin, and it means most large organizations already run routing somewhere whether or not anyone approved it. Blind spot: nobody outside the team knows the policy exists.

Procurement-led. The router arrives bundled inside a cloud contract, as Bedrock Intelligent Prompt Routing or the Foundry Model Router. It requires no new vendor and no new review, which is exactly why it spreads. Blind spot: your model pool was chosen by your cloud provider, and the routing policy is governed by a contract rather than by a policy you wrote.

Advisory-led. A systems integrator brings routing in as part of a cost-reduction program, often with the savings committed before the work starts. Accenture Ventures has taken a position in the routing market directly. Blind spot: when the saving is pre-committed, quality becomes the variable that flexes.

None of the three begins with a quality baseline. All three begin with a cost target. That is the single most useful thing to notice about how this technology is being adopted.

What good looks like

Five disciplines separate the organizations getting real savings from the ones quietly buying degradation.

1. Baseline before you route. You cannot detect a regression from a state you never measured. Capture answer quality on representative traffic while everything still runs on one model.
2. Gate every routing change on an eval suite. Fifty to five hundred representative cases, run in CI, blocking the change. Treat a routing policy change with the same discipline as a database schema migration, because it has a comparable blast radius and considerably worse detectability.
3. Tier the workloads explicitly. Some decisions are pinned to a named model regardless of price. That list should be short, written down, and owned by someone with a title. Everything not on it is routable by default.
4. Attribute in production. Every response carries the model that produced it. Without that, a complaint cannot be traced back to a routing decision, and the feedback loop that would correct the policy never closes.
5. Put a name on the policy. Someone should own the quality-per-dollar setting the way someone owns pricing. Not a committee, and not whatever the platform team defaulted to.

The organizations getting this right treat routing as a quality program with a cost benefit, rather than a cost program with a quality risk. Same activity, opposite governance, and a very different result two years out.

III. What the Router Does Not Do

A router is a dispatcher, not a librarian.

By the time it makes a decision, the request is already assembled: system prompt, retrieved documents, conversation history, tool definitions. The router picks a destination for that payload. It does not fetch, rank, or trim anything. Same context, different model.

Which means the first objection to everything above, that a cheaper model will give worse answers, is usually not a model problem at all.

A frontier model compensates for mediocre retrieval using its own parametric knowledge. Give it fifty loosely relevant paragraphs and it still finds the answer, partly because it already half knew it. A smaller model answers from just what you gave it. So a large share of "the cheap model was worse" findings are really "our retrieval was always weak and the expensive model was hiding it." Routing does not create that problem. It removes the concealment.

A cheap model with the right five paragraphs beats a frontier model with the wrong fifty. That is the whole game, and it is decided one layer below the router.

Three practical constraints follow, and they are the ones I would have an architect answer before approving a routing program.

The context window collapses to the smallest model in the pool. Microsoft states this plainly in its Foundry router documentation: the effective context window is limited by the smallest underlying model. Either constrain the pool to models that meet your requirement, or retrieve more selectively so the payload is smaller to begin with.

Prompts are not portable. A prompt tuned against one model degrades on another, and the failure is usually format compliance and tool calling rather than knowledge. Per-tier prompt templates are table stakes, not an optimization.

Caching cuts the other way from what you would expect. The instinct is that routing across providers destroys prompt caching and the savings with it. Benchmarked, the opposite holds: sessions return to a model whose cache is still warm 99.3% of the time, and routing with caching enabled came out 37% to 69% cheaper than caching a single frontier model. But run a router with caching switched off and it is roughly four times more expensive than simply caching one fixed model. The router is not the risk. Turning off the cache underneath it is.

The strategic consequence sits one layer down. The cheaper your models get, the more of your answer quality is carried by retrieval, and the less any model can paper over. Routing at the control plane raises the stakes in the data and context layer. Anyone treating retrieval as a solved problem is about to discover it was being subsidized by frontier pricing.

IV. What the Data Already Shows

The aggregate effect is visible in production telemetry, and it moved faster than most forecasts.

Vercel's AI Gateway Production Index for June 2026 reported that open-weight models handled 29% of all tokens on the platform while accounting for under 4% of total spending. Two months earlier, open weights were around 11% of tokens. Open models on that platform run at roughly one tenth the average cost per token. Vercel's head of agentic infrastructure put it plainly: "Price is doing the work here. When a task doesn't need the best model, teams are beginning to route it to the cheapest one."

The same index shows the other half of the picture. Anthropic captured 61% of platform spending on 32% of tokens. The premium tier is not collapsing. It is concentrating.

On OpenRouter, US-origin models fell from roughly 70% of token volume to about 30% between June 2025 and June 2026, per data compiled by Bloomberg and Exponential View. The July 2026 State of Open Source AI report puts Chinese-origin models at 46.4% of routed tokens against 35.7% from US providers, with the seven highest-volume models on the platform all shipping open weights.

One caveat worth stating plainly, because it is where most commentary overreaches: these platforms skew toward cost-sensitive developers, not regulated enterprise buyers. They are a leading indicator, not a census. But the direction has been consistent for four consecutive quarters, and enterprise spending data has begun to follow.

V. Implications for Open Systems

The conventional read is that open-weight models won on price. I think that is half right and misses the mechanism.

Open models did not win on price alone. Cheap alternatives have existed throughout. What changed is that the router made them substitutable. Before routing infrastructure matured, adopting a second model meant a second integration, a second prompt library, a second eval harness, and a second set of failure modes. That friction was the frontier labs' real moat, and it was worth more than any benchmark lead.

The router dissolves it. When model selection is a configuration decision rather than an architectural one, switching cost approaches zero, and a market with no switching cost prices at marginal cost on every workload where capability is not the binding constraint.

The strongest open model now sits fourth overall on the Artificial Analysis Intelligence Index, roughly one release cycle behind the closed frontier. One release cycle is not a moat. It is a scheduling problem.

Two constraints keep this from being a rout, and both matter for anyone making a build decision this year.

First, deployment. The State of Open Source AI report found that 79% of developers adding AI functionality use open models, but only 53% of open-model teams reach production, against 63% for closed. Open ships easily and deploys hard. The cost of self-hosting, securing, and governing an open model is real and is routinely omitted from the comparison that produced the "one tenth the price" headline.

Second, accountability. When a closed frontier model produces a harmful output, there is a vendor, a contract, and an indemnity. When a self-hosted open model does it, there is you. For a regulated enterprise, that difference is worth a substantial premium, and it is the most durable thing the frontier labs are selling.

VI. Implications for the Frontier Labs

Two counter-moves are already in market.

Internalize the router. OpenAI's GPT-5 Auto routes each request between GPT-5 and GPT-5 Instant based on prompt complexity, at no charge beyond token costs, with enterprise customers able to set a cheaper default. Microsoft's Foundry Model Router does the same across a pool of models inside Azure. The logic is sound. If the cheap-versus-expensive arbitrage is going to be captured by somebody, it should be captured inside your own stack rather than by a third-party gateway that also lists your competitors. A vendor-native router is a retention product wearing a cost-savings label.

Retreat up-stack on purpose. The alternative is to accept the role of escalation tier: stop competing for the 80% of traffic that routers will send elsewhere regardless, and price the hard 20% that only you can serve. That is a smaller, higher-margin, more defensible business. It is also not the business that the current infrastructure buildout is underwriting, which is the tension worth watching over the next several quarters. Note that closed frontier pricing rose roughly 12% on that same Vercel index even as cheap alternatives proliferated. That is not a market under price pressure. That is a market bifurcating.

There is a third move, less discussed and probably the most important: own the router as a control point. Whoever operates the router sees every prompt, every escalation, every quality signal, and every dollar. That is the richest evaluation dataset in the enterprise and the closest thing to a durable customer relationship in a stack where the models themselves are becoming fungible. The router looks like plumbing. It is positioning.

VII. The Governance Question

For boards and operating executives, this is the part that matters most, and it is the part that is currently unowned.

A routing policy is a business decision expressed as a classifier. It sets the quality-per-dollar tradeoff on every customer interaction the company has. It determines whether your highest-value client's question was answered by a frontier model or by a two-cent alternative. It is, in a meaningful sense, a pricing and service-level decision made continuously, at machine speed, by a component that was likely deployed by a platform team optimizing a cloud bill.

Six questions worth asking in your next technology review:

1. Do we route? If so, what percentage of traffic reaches the frontier tier, and who set that number?
2. What eval suite gates a routing change, and when was it last refreshed against real production traffic?
3. Can we attribute any given customer interaction to the model that produced it?
4. Which decisions are hard-pinned to a specific model regardless of cost, and is that list written down?
5. Is prompt caching on, and does our routing policy preserve it? A router with the cache switched off costs more than not routing at all.
6. If our primary model vendor doubled its price tomorrow, how many days to reroute?

That last one is the practical takeaway. Routing is not only a cost tactic. It is negotiating position. Enterprises that architect for substitutability are the ones that will hold pricing power in this market, and the labs know it. That is precisely why they would prefer to run the router themselves.

Christopher C. Flaesch is Managing Principal of Echelon Gateway Advisors LLC, advising enterprise technology leaders on AI value capture, go-to-market strategy, and partner ecosystems.