What is an AI gateway? It's a single endpoint between your code and the model providers you call, such as OpenAI, Anthropic and Google. Every LLM request goes through it, and the gateway checks the key, picks a model and provider, retries or falls back when a call fails, answers repeats from a cache, logs tokens and cost, and stops spending at a limit you set.
Think of it as an API gateway that counts tokens instead of requests. Microsoft builds its AI gateway into Azure API Management, and the headline policies are token limits, token metrics and semantic caching. Hosted gateways such as OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway and Requesty run the layer for you; open-source proxies such as LiteLLM run on your own servers.
Below: the six jobs, the four kinds of gateway, how they charge, what BYOK means, how routers pick a model, and when going direct is better. Prices and terms were checked on each company's pages on 5 November 2026.
What is an AI gateway, and what does it do?
An AI gateway does six jobs. Most products do all six; they differ in how far each one goes.

| Job | What it does | Example |
|---|---|---|
| One key, one API | Your code calls one endpoint, usually in OpenAI's Chat Completions format, and the gateway translates for each provider | Change the base URL and the model string; the rest of the code stays |
| Routing | Chooses the provider for a model, or the model for a request | OpenRouter weighs price, throughput, latency and error rates across each model's providers |
| Fallbacks | On an error, a 429 or an outage, retries on another provider or model | OpenRouter's models list tries the next model when one fails |
| Caching | Returns a stored answer to a repeated request without calling the model | Cloudflare needs an exact match; Azure API Management matches by meaning |
| Logging | Records model, tokens, cost, latency and sometimes the prompt and reply | Cloudflare logs by default; Vercel deletes prompt content when the request completes |
| Spend controls | Caps tokens or dollars per key, team, app or user | Azure's llm-token-limit sets tokens per minute or a quota per period |
Gateway caching isn't the same as a provider's prompt caching. A gateway cache skips the model call, but only on a repeat. Cloudflare's cache needs the entire request to match, is off by default, and keeps entries from 60 seconds to one month; Azure API Management also matches similar prompts. Prompt caching is a discount the provider gives on a repeated prefix: on Claude Sonnet 5.5, a cache hit costs $0.10 per million tokens against $2.00 for fresh input. For chat and agent traffic, the provider's cache usually saves more, and a gateway that moves traffic between models can throw it away.
Logging is a data question. Cloudflare logs every request by default, and each log "can include the prompt, response, provider"; send cf-aig-collect-log-payload: false to keep metadata only. The zero data retention guide covers what the providers behind a gateway keep.
The four kinds of AI gateway
| Kind | Who holds the provider keys | Who bills you for tokens | Examples |
|---|---|---|---|
| Hosted, pooled keys | The gateway | The gateway, from a prepaid balance | OpenRouter, Requesty, Vercel AI Gateway, Eden AI, Cloudflare Unified Billing, Aight |
| Hosted, your keys | You, stored in the gateway | Each provider | Cloudflare AI Gateway with stored keys, Portkey, BYOK on OpenRouter or Vercel |
| Self-hosted | You | Each provider | LiteLLM, Portkey's open-source gateway |
| Cloud API management | You, often through managed identity | Your cloud bill | Azure API Management |
The pooled kind is what people usually mean by one API key for every model: one account and one balance instead of one per maker. The one-key setup guide walks through it. The other three are control layers in front of provider accounts you already hold.
Self-hosting is free software, not free infrastructure. LiteLLM's own benchmark puts its added latency at 2 ms median and 8 ms at p95 on four 4-vCPU instances serving 1,170 requests a second, and 12 ms and 29 ms on two instances. You also own upgrades: LiteLLM shipped v1.108.0 on 4 November.
How AI gateways charge
Almost every gateway bills tokens at the model maker's list price, so the cost is in the fees around them. There are five patterns:
- A fee on each purchase: OpenRouter 5.5% by card ($0.80 minimum) or 5% in crypto, Eden AI 5.5%, Requesty 5%, Cloudflare Unified Billing 5%.
- No token fee: Vercel adds no markup but passes card processing fees to you.
- A BYOK fee: OpenRouter's is zero on the first $25,000 of list-price usage a month, then 5%. Vercel and Cloudflare list none.
- A plan: Portkey Production is $49 a month for 100,000 logged requests, plus $9 per extra 100,000, and providers bill the tokens.
- Add-ons: Vercel charges $0.10 per 1,000 requests for team-wide zero data retention, and the same for a team-wide provider allowlist.

| Option | How it charges | $1,000 of list-price usage costs |
|---|---|---|
| OpenRouter, card | 5.5% per purchase, $0.80 minimum | $1,055 |
| Eden AI | 5.5% per purchase | $1,055 |
| Requesty | 5% per purchase | $1,050 |
| Cloudflare Unified Billing | 5% per purchase | $1,050 |
| Vercel AI Gateway | No token fee | $1,000 plus card processing |
| Direct provider account | None | $1,000 |
| Portkey Production | $49 plan; providers bill tokens | $1,049 |
| Self-hosted gateway | None | $1,000 plus servers and upkeep |
| Aight, outside India | 3% processing, 10% extra balance | $936 |
Minimums matter at small sizes: OpenRouter's $0.80 minimum fee makes a $5 card purchase cost $5.80, 16% over list. The aggregator price comparison works through purchase sizes, refunds and expiry, and OpenRouter alternatives compares the terms side by side.
Two costs never show up as a fee. OpenAI and Anthropic charge 50% less through their Batch APIs for work that can wait up to 24 hours, so check that a gateway has a batch route before moving batch jobs to it. And region pinning costs extra everywhere: Anthropic charges 1.1x for US-only inference, and Claude on Bedrock and Google Cloud costs 10% more on regional endpoints, whichever gateway sends the request.
BYOK meaning: bring your own key
BYOK means bring your own key: you give the gateway your own provider API key, so the provider bills your account directly and the gateway only routes, logs and limits the traffic.
The acronym means something else in cloud security, where BYOK is importing an encryption key you generated into a cloud key vault. Microsoft uses it for moving a key from an on-premises HSM into Azure Key Vault. In AI gateways and coding tools, it means a model provider's API key.
How three hosted gateways treat it:
| Gateway | Fee on BYOK traffic | Fallback to the gateway's own keys |
|---|---|---|
| OpenRouter | Free on the first $25,000 of list-price usage a month, then 5% of the normal OpenRouter price | On by default when your keys fail or hit limits, billed to your balance; turn off with "Never use shared capacity" or is_byok_only |
| Vercel AI Gateway | None, but BYOK needs a purchased balance | Failed requests retry on Vercel's keys, billed to your balance |
| Cloudflare AI Gateway | None listed | Requests without your key go to Unified Billing unless "Require provider credentials" is on |
What you save is the purchase fee, plus anything your provider contract gives you that a gateway can't: committed-spend discounts, startup program allowances, higher rate limits and your own data agreements. On OpenRouter, a team spending $40,000 a month at list price pays $750 in BYOK fees, against $2,200 in purchase fees for the same usage. Under $25,000 a month, BYOK there is free.
What you give up:
- One bill. You keep an account, a balance and a rate-limit tier with every provider.
- A hard ceiling. A failing key still spends through the gateway's keys, and Vercel's spend limits don't count BYOK traffic, so set limits at each provider too.
- Some privacy routing. Vercel's zero-data-retention routing skips BYOK keys unless you mark a key as covered by your own provider agreement.
LLM routers and automatic routing
An LLM router, also sold as an API router or model router, decides where each request goes. It makes two separate decisions, and products blur them:
- Provider routing sends the same model to a different host. OpenRouter's default compares effective price, throughput, time to first token, rate limits, reliability and quality for each provider of a model, then picks one at random, weighted by those scores.
- Model routing sends each request to a different model. Rules you write (classification to a small model, code review to a large one) stay predictable. Automatic routing hands the choice to a classifier or a trained model.
How automatic routers choose
| Router | How it picks | What you control | Charge for routing |
|---|---|---|---|
| OpenRouter Auto Router | A classifier assigns one of about 30 task types, then ranks models by their share of OpenRouter spend on that task over the last 7 days | cost_tier from low to max; allowed and excluded models | None |
| Microsoft Foundry model router | A trained model scores each prompt; Balanced mode takes the lowest-cost model within about 1% to 2% of the best quality, Cost mode within 5% to 6% | Mode and a model subset | Input prompts billed at the router's listed rate |
| Amazon Bedrock intelligent prompt routing | Predicts response quality for exactly two models from one family | Fallback model and a quality-difference threshold | Not stated on the pages checked |
OpenRouter returns the chosen model in the response's model field, and AWS includes the model used in its response. Log it on every request.
When automatic routing saves money
Routing pays when most requests are easy, each one stands alone, and the price gap between models is wide. Take one million support requests a month at 2,000 input and 300 output tokens each, with no caching. On Claude Sonnet 5.5 at $2 / $10 per million tokens, that's $7,000. If 70% can go to Claude Haiku 5.5 at $0.10 / $0.50, the month costs $2,345, 66% less.
The research points the same way for the same kind of traffic. LMSYS reported that RouteLLM cut costs by over 85% on MT Bench, 45% on MMLU and 35% on GSM8K against GPT-4 alone, keeping 95% of its quality, with GPT-4 Turbo as the strong model and Mixtral 8x7B as the weak one (1 July 2024). AWS says its router "can reduce costs by up to 30% without compromising on accuracy." Both figures come from benchmark prompts, not your traffic.
When automatic routing hides costs
- The prompt cache resets when the model changes. An agent that resends 100,000 tokens of context each step pays $0.02 to read it from Claude Opus 5.5's cache. Move the next step to Sonnet 5.5, half the price, and that step pays $0.25 to write the context into Sonnet's cache. At 1,000 output tokens a step, Sonnet needs 11 more steps to earn the switch back. Microsoft says caching helps only when the same model handles consecutive requests.
- Fallbacks bill whatever answered. OpenRouter prices a request using the model that was ultimately used, so a free model that falls back to a paid one costs the paid rate.
- Cost per request stops being fixed, so size spend caps for the priciest model in the pool.
- Limits shrink to the weakest model: Microsoft's router takes the smallest context window of its models.
- The ranking isn't yours. OpenRouter's Auto Router ranks by what other customers spend on a task type, not by results on your data, and your evals have to cover every model it can pick.
If you route automatically, keep a session on one model, log the model on every response, and compare a week of routed traffic with a single-model baseline first.
When you need an AI gateway, and when to go direct

A gateway earns its place when:
- You run two or more providers in production and want failover between them.
- Several apps or teams share providers, and each needs its own key, spend cap and cost report.
- You want one bill and one balance instead of an account per maker.
- You're testing many models and want to swap them without code changes.
- Security or compliance wants one place to enforce logging, content checks and allowed models.
Going direct is better when:
- Most of your spend goes to one provider. A purchase-fee gateway adds 5% to 5.5%, and new provider features can reach gateways later.
- The work can wait. Batch halves the price at OpenAI and Anthropic.
- Agents depend on long cached sessions on one model.
- You hold negotiated discounts or committed spend. BYOK or a self-hosted gateway also keeps them.
- You need the shortest data path, because every intermediary sees your prompts and keys.
That last point isn't theoretical. In a 9 April 2026 study, researchers bought or collected 428 third-party LLM routers and found 9 injecting malicious code and 17 touching AWS credentials planted as bait. A gateway terminates TLS and reads every request in plain text, so choose one whose operator, terms and data handling you'd accept for your keys.
Going direct doesn't mean giving up failover. The basic version fits in the client: try the primary route, and move on after a timeout, a 429, a 5xx or a retired model, but not after a bad request, a bad key or an empty balance. This works with the OpenAI Python SDK 3.45.0 against any two OpenAI-compatible endpoints:
import os
from openai import OpenAI, APIStatusError, APIConnectionError
# Each route: an OpenAI-compatible endpoint, its key, and a model ID it serves.
ROUTES = [
(OpenAI(base_url=os.environ["PRIMARY_BASE_URL"], api_key=os.environ["PRIMARY_KEY"]),
os.environ["PRIMARY_MODEL"]),
(OpenAI(base_url=os.environ["BACKUP_BASE_URL"], api_key=os.environ["BACKUP_KEY"]),
os.environ["BACKUP_MODEL"]),
]
NEXT_ROUTE = {404, 408, 409, 429} # plus every 5xx and connection error
def complete(messages):
last_error = None
for client, model in ROUTES:
try:
r = client.chat.completions.create(model=model, messages=messages)
return model, r.choices[0].message.content, r.usage
except APIStatusError as e:
if e.status_code < 500 and e.status_code not in NEXT_ROUTE:
raise # bad request, bad key or empty balance: the next route won't fix it
last_error = e
except APIConnectionError as e:
last_error = e
raise last_error
The SDK already retries 408, 409, 429 and 5xx responses twice with backoff, so the loop moves on only after a route has really failed. Log the returned model and usage; the monthly LLM cost guide turns usage into a bill.
Where Aight fits
Aight is a hosted gateway of the pooled-key kind: one OpenAI-compatible key for 138 models from 16 makers, paid from one prepaid US dollar balance, with most models at the maker's list price. Outside India, a $100 recharge costs $103 with 3% processing and adds $110 of balance, so $1,000 of list-price usage costs about $936, less than the purchase-fee gateways above.
Its controls sit on each key: a monthly cap that returns 403 key_cap_reached, an expiry date and a list of allowed models. Prompts and outputs aren't stored unless you turn on 7-day debug logging.
For this decision, what it doesn't do matters as much. Aight documents three endpoints (POST /v1/chat/completions, POST /v1/embeddings and GET /v1/models) and none of the gateway features in the routing section: no BYOK, no automatic routing, no gateway-side fallbacks or caching, no batch route. With one key, a fallback is a second model ID in a client loop like the one above. If you hold negotiated provider discounts, BYOK on another gateway or a self-hosted gateway keeps them, and if most of your spend is batch work or one provider, going direct costs less. Compare each model's price on the models page with its maker's before moving traffic.
What developers report
- "managing that extra layer of uncertainty can cost more than it saves" (Manifest, on shutting down the LLM router in its gateway after four months and 7,000 cloud users, 31 Jul 2026, link)
- "Routing belongs on the client side" (a Hacker News commenter on the Manifest post, 31 Jul 2026, link)
- "You cant fail over to a different model though." (a Hacker News commenter, who added that switching providers isn't automatic either, citing Bedrock and Anthropic's tool search tool, 31 Jul 2026, link)
- The team behind Weave's coding-agent router said routers that pick each request independently spend more on agents through cache misses, and that its cache-aware router cut its own spend by 40% (Show HN, 26 Jun 2026, link)
- Of the 28 paid and 400 free routers tested, one drained ether from a private key the researchers had planted (arXiv paper, 9 Apr 2026, link)
FAQ
What is the difference between an API gateway and an AI gateway?
An API gateway manages traffic to your own services and counts requests. An AI gateway manages calls out to model providers and counts tokens, so its limits, metrics and caching work on tokens and prompts. Microsoft's AI gateway is an extension of the Azure API Management gateway, not a separate product.
Is OpenRouter an AI gateway?
Yes, of the hosted, pooled-key kind. Its pricing page lists 600+ models from 80+ providers behind one key and one balance, with provider routing, model fallbacks and an Auto Router. It charges 5.5% on card purchases, and BYOK is free on the first $25,000 of list-price usage a month.
Does an AI gateway add latency?
A little. A hosted gateway adds a network hop, and a self-hosted one adds processing time: LiteLLM's benchmark measured 2 ms median and 8 ms at p95 on four instances. Retries and fallbacks add more when they trigger, but they replace an error you would otherwise return.
What does BYOK mean in AI tools?
Bring your own key: the tool or gateway uses your own provider API key, so the provider bills you directly. Coding tools and gateways use the term the same way. In cloud security, BYOK means importing your own encryption key into a provider's key vault.
Do AI gateways store prompts?
It depends on the gateway and its settings. Cloudflare logs by default and its logs can include prompts, Vercel deletes prompt content when a request completes, and OpenRouter doesn't log prompts unless you opt in. The provider behind any gateway has its own retention policy too.
Sources
- Microsoft: AI gateway capabilities in Azure API Management, Model router for Microsoft Foundry, Bring your own key specification for Azure Key Vault
- Cloudflare: AI Gateway overview, Caching, Logging, Pricing, Unified Billing, Bring your own keys
- Vercel: AI Gateway pricing, BYOK, AI Gateway FAQ
- OpenRouter: Pricing, FAQ, BYOK, Provider routing, Model fallbacks, Auto Router
- Requesty, Pricing; Eden AI, Pricing; Portkey, Pricing and gateway on GitHub
- LiteLLM: GitHub repository and releases, Benchmarks
- Amazon Web Services: Intelligent prompt routing docs, Intelligent Prompt Routing
- Anthropic, Pricing; OpenAI, Batch API; OpenAI Python SDK on PyPI
- Research: Ong et al., RouteLLM and the LMSYS RouteLLM post; Liu et al., Your Agent Is Mine: Measuring Malicious Intermediary Attacks on the LLM Supply Chain
- Manifest, post on deprecating its LLM router, 31 July 2026
- Hacker News: Manifest post discussion, Show HN: Weave Router
- Aight, Models and pricing, Quickstart, API reference, Refund policy, Privacy
Checked on 5 November 2026. Updated 17 November 2026: Aight's current model list.