Blog

What is an AI gateway? What it does, how it charges, BYOK, LLM routers, and when to go direct

Ramashankar S · · Updated · 17 min read

What is an AI gateway? It's a single endpoint between your code and the model providers you call, such as OpenAI, Anthropic and Google. Every LLM request goes through it, and the gateway checks the key, picks a model and provider, retries or falls back when a call fails, answers repeats from a cache, logs tokens and cost, and stops spending at a limit you set.

Think of it as an API gateway that counts tokens instead of requests. Microsoft builds its AI gateway into Azure API Management, and the headline policies are token limits, token metrics and semantic caching. Hosted gateways such as OpenRouter, Vercel AI Gateway, Cloudflare AI Gateway and Requesty run the layer for you; open-source proxies such as LiteLLM run on your own servers.

Below: the six jobs, the four kinds of gateway, how they charge, what BYOK means, how routers pick a model, and when going direct is better. Prices and terms were checked on each company's pages on 5 November 2026.

What is an AI gateway, and what does it do?

An AI gateway does six jobs. Most products do all six; they differ in how far each one goes.

How a request moves through an AI gateway: key and cap check, cache, routing, provider call with a pooled key or your own key, fallback on errors, and logging
JobWhat it doesExample
One key, one APIYour code calls one endpoint, usually in OpenAI's Chat Completions format, and the gateway translates for each providerChange the base URL and the model string; the rest of the code stays
RoutingChooses the provider for a model, or the model for a requestOpenRouter weighs price, throughput, latency and error rates across each model's providers
FallbacksOn an error, a 429 or an outage, retries on another provider or modelOpenRouter's models list tries the next model when one fails
CachingReturns a stored answer to a repeated request without calling the modelCloudflare needs an exact match; Azure API Management matches by meaning
LoggingRecords model, tokens, cost, latency and sometimes the prompt and replyCloudflare logs by default; Vercel deletes prompt content when the request completes
Spend controlsCaps tokens or dollars per key, team, app or userAzure's llm-token-limit sets tokens per minute or a quota per period

Gateway caching isn't the same as a provider's prompt caching. A gateway cache skips the model call, but only on a repeat. Cloudflare's cache needs the entire request to match, is off by default, and keeps entries from 60 seconds to one month; Azure API Management also matches similar prompts. Prompt caching is a discount the provider gives on a repeated prefix: on Claude Sonnet 5.5, a cache hit costs $0.10 per million tokens against $2.00 for fresh input. For chat and agent traffic, the provider's cache usually saves more, and a gateway that moves traffic between models can throw it away.

Logging is a data question. Cloudflare logs every request by default, and each log "can include the prompt, response, provider"; send cf-aig-collect-log-payload: false to keep metadata only. The zero data retention guide covers what the providers behind a gateway keep.

What an AI gateway does, and when routing backfiresDevsplainers · 11:21 · YouTube
A gateway's jobs, rule-based and learned routing, and why a model switch can cost more once the cache is lost.

The four kinds of AI gateway

KindWho holds the provider keysWho bills you for tokensExamples
Hosted, pooled keysThe gatewayThe gateway, from a prepaid balanceOpenRouter, Requesty, Vercel AI Gateway, Eden AI, Cloudflare Unified Billing, Aight
Hosted, your keysYou, stored in the gatewayEach providerCloudflare AI Gateway with stored keys, Portkey, BYOK on OpenRouter or Vercel
Self-hostedYouEach providerLiteLLM, Portkey's open-source gateway
Cloud API managementYou, often through managed identityYour cloud billAzure API Management

The pooled kind is what people usually mean by one API key for every model: one account and one balance instead of one per maker. The one-key setup guide walks through it. The other three are control layers in front of provider accounts you already hold.

Self-hosting is free software, not free infrastructure. LiteLLM's own benchmark puts its added latency at 2 ms median and 8 ms at p95 on four 4-vCPU instances serving 1,170 requests a second, and 12 ms and 29 ms on two instances. You also own upgrades: LiteLLM shipped v1.108.0 on 4 November.

How AI gateways charge

Almost every gateway bills tokens at the model maker's list price, so the cost is in the fees around them. There are five patterns:

  1. A fee on each purchase: OpenRouter 5.5% by card ($0.80 minimum) or 5% in crypto, Eden AI 5.5%, Requesty 5%, Cloudflare Unified Billing 5%.
  2. No token fee: Vercel adds no markup but passes card processing fees to you.
  3. A BYOK fee: OpenRouter's is zero on the first $25,000 of list-price usage a month, then 5%. Vercel and Cloudflare list none.
  4. A plan: Portkey Production is $49 a month for 100,000 logged requests, plus $9 per extra 100,000, and providers bill the tokens.
  5. Add-ons: Vercel charges $0.10 per 1,000 requests for team-wide zero data retention, and the same for a team-wide provider allowlist.
How AI gateways charge, and what $1,000 of list-price usage costs under each model
OptionHow it charges$1,000 of list-price usage costs
OpenRouter, card5.5% per purchase, $0.80 minimum$1,055
Eden AI5.5% per purchase$1,055
Requesty5% per purchase$1,050
Cloudflare Unified Billing5% per purchase$1,050
Vercel AI GatewayNo token fee$1,000 plus card processing
Direct provider accountNone$1,000
Portkey Production$49 plan; providers bill tokens$1,049
Self-hosted gatewayNone$1,000 plus servers and upkeep
Aight, outside India3% processing, 10% extra balance$936

Minimums matter at small sizes: OpenRouter's $0.80 minimum fee makes a $5 card purchase cost $5.80, 16% over list. The aggregator price comparison works through purchase sizes, refunds and expiry, and OpenRouter alternatives compares the terms side by side.

Two costs never show up as a fee. OpenAI and Anthropic charge 50% less through their Batch APIs for work that can wait up to 24 hours, so check that a gateway has a batch route before moving batch jobs to it. And region pinning costs extra everywhere: Anthropic charges 1.1x for US-only inference, and Claude on Bedrock and Google Cloud costs 10% more on regional endpoints, whichever gateway sends the request.

BYOK meaning: bring your own key

BYOK means bring your own key: you give the gateway your own provider API key, so the provider bills your account directly and the gateway only routes, logs and limits the traffic.

The acronym means something else in cloud security, where BYOK is importing an encryption key you generated into a cloud key vault. Microsoft uses it for moving a key from an on-premises HSM into Azure Key Vault. In AI gateways and coding tools, it means a model provider's API key.

How three hosted gateways treat it:

GatewayFee on BYOK trafficFallback to the gateway's own keys
OpenRouterFree on the first $25,000 of list-price usage a month, then 5% of the normal OpenRouter priceOn by default when your keys fail or hit limits, billed to your balance; turn off with "Never use shared capacity" or is_byok_only
Vercel AI GatewayNone, but BYOK needs a purchased balanceFailed requests retry on Vercel's keys, billed to your balance
Cloudflare AI GatewayNone listedRequests without your key go to Unified Billing unless "Require provider credentials" is on

What you save is the purchase fee, plus anything your provider contract gives you that a gateway can't: committed-spend discounts, startup program allowances, higher rate limits and your own data agreements. On OpenRouter, a team spending $40,000 a month at list price pays $750 in BYOK fees, against $2,200 in purchase fees for the same usage. Under $25,000 a month, BYOK there is free.

What you give up:

  • One bill. You keep an account, a balance and a rate-limit tier with every provider.
  • A hard ceiling. A failing key still spends through the gateway's keys, and Vercel's spend limits don't count BYOK traffic, so set limits at each provider too.
  • Some privacy routing. Vercel's zero-data-retention routing skips BYOK keys unless you mark a key as covered by your own provider agreement.

LLM routers and automatic routing

An LLM router, also sold as an API router or model router, decides where each request goes. It makes two separate decisions, and products blur them:

  • Provider routing sends the same model to a different host. OpenRouter's default compares effective price, throughput, time to first token, rate limits, reliability and quality for each provider of a model, then picks one at random, weighted by those scores.
  • Model routing sends each request to a different model. Rules you write (classification to a small model, code review to a large one) stay predictable. Automatic routing hands the choice to a classifier or a trained model.

How automatic routers choose

RouterHow it picksWhat you controlCharge for routing
OpenRouter Auto RouterA classifier assigns one of about 30 task types, then ranks models by their share of OpenRouter spend on that task over the last 7 dayscost_tier from low to max; allowed and excluded modelsNone
Microsoft Foundry model routerA trained model scores each prompt; Balanced mode takes the lowest-cost model within about 1% to 2% of the best quality, Cost mode within 5% to 6%Mode and a model subsetInput prompts billed at the router's listed rate
Amazon Bedrock intelligent prompt routingPredicts response quality for exactly two models from one familyFallback model and a quality-difference thresholdNot stated on the pages checked

OpenRouter returns the chosen model in the response's model field, and AWS includes the model used in its response. Log it on every request.

When automatic routing saves money

Routing pays when most requests are easy, each one stands alone, and the price gap between models is wide. Take one million support requests a month at 2,000 input and 300 output tokens each, with no caching. On Claude Sonnet 5.5 at $2 / $10 per million tokens, that's $7,000. If 70% can go to Claude Haiku 5.5 at $0.10 / $0.50, the month costs $2,345, 66% less.

The research points the same way for the same kind of traffic. LMSYS reported that RouteLLM cut costs by over 85% on MT Bench, 45% on MMLU and 35% on GSM8K against GPT-4 alone, keeping 95% of its quality, with GPT-4 Turbo as the strong model and Mixtral 8x7B as the weak one (1 July 2024). AWS says its router "can reduce costs by up to 30% without compromising on accuracy." Both figures come from benchmark prompts, not your traffic.

When automatic routing hides costs

  • The prompt cache resets when the model changes. An agent that resends 100,000 tokens of context each step pays $0.02 to read it from Claude Opus 5.5's cache. Move the next step to Sonnet 5.5, half the price, and that step pays $0.25 to write the context into Sonnet's cache. At 1,000 output tokens a step, Sonnet needs 11 more steps to earn the switch back. Microsoft says caching helps only when the same model handles consecutive requests.
  • Fallbacks bill whatever answered. OpenRouter prices a request using the model that was ultimately used, so a free model that falls back to a paid one costs the paid rate.
  • Cost per request stops being fixed, so size spend caps for the priciest model in the pool.
  • Limits shrink to the weakest model: Microsoft's router takes the smallest context window of its models.
  • The ranking isn't yours. OpenRouter's Auto Router ranks by what other customers spend on a task type, not by results on your data, and your evals have to cover every model it can pick.

If you route automatically, keep a session on one model, log the model on every response, and compare a week of routed traffic with a single-model baseline first.

When you need an AI gateway, and when to go direct

Decision guide: when an AI gateway earns its place and when calling providers directly is better

A gateway earns its place when:

  • You run two or more providers in production and want failover between them.
  • Several apps or teams share providers, and each needs its own key, spend cap and cost report.
  • You want one bill and one balance instead of an account per maker.
  • You're testing many models and want to swap them without code changes.
  • Security or compliance wants one place to enforce logging, content checks and allowed models.

Going direct is better when:

  • Most of your spend goes to one provider. A purchase-fee gateway adds 5% to 5.5%, and new provider features can reach gateways later.
  • The work can wait. Batch halves the price at OpenAI and Anthropic.
  • Agents depend on long cached sessions on one model.
  • You hold negotiated discounts or committed spend. BYOK or a self-hosted gateway also keeps them.
  • You need the shortest data path, because every intermediary sees your prompts and keys.

That last point isn't theoretical. In a 9 April 2026 study, researchers bought or collected 428 third-party LLM routers and found 9 injecting malicious code and 17 touching AWS credentials planted as bait. A gateway terminates TLS and reads every request in plain text, so choose one whose operator, terms and data handling you'd accept for your keys.

Going direct doesn't mean giving up failover. The basic version fits in the client: try the primary route, and move on after a timeout, a 429, a 5xx or a retired model, but not after a bad request, a bad key or an empty balance. This works with the OpenAI Python SDK 3.45.0 against any two OpenAI-compatible endpoints:

import os
from openai import OpenAI, APIStatusError, APIConnectionError

# Each route: an OpenAI-compatible endpoint, its key, and a model ID it serves.
ROUTES = [
    (OpenAI(base_url=os.environ["PRIMARY_BASE_URL"], api_key=os.environ["PRIMARY_KEY"]),
     os.environ["PRIMARY_MODEL"]),
    (OpenAI(base_url=os.environ["BACKUP_BASE_URL"], api_key=os.environ["BACKUP_KEY"]),
     os.environ["BACKUP_MODEL"]),
]
NEXT_ROUTE = {404, 408, 409, 429}  # plus every 5xx and connection error


def complete(messages):
    last_error = None
    for client, model in ROUTES:
        try:
            r = client.chat.completions.create(model=model, messages=messages)
            return model, r.choices[0].message.content, r.usage
        except APIStatusError as e:
            if e.status_code < 500 and e.status_code not in NEXT_ROUTE:
                raise  # bad request, bad key or empty balance: the next route won't fix it
            last_error = e
        except APIConnectionError as e:
            last_error = e
    raise last_error

The SDK already retries 408, 409, 429 and 5xx responses twice with backoff, so the loop moves on only after a route has really failed. Log the returned model and usage; the monthly LLM cost guide turns usage into a bill.

Where Aight fits

Aight is a hosted gateway of the pooled-key kind: one OpenAI-compatible key for 138 models from 16 makers, paid from one prepaid US dollar balance, with most models at the maker's list price. Outside India, a $100 recharge costs $103 with 3% processing and adds $110 of balance, so $1,000 of list-price usage costs about $936, less than the purchase-fee gateways above.

Its controls sit on each key: a monthly cap that returns 403 key_cap_reached, an expiry date and a list of allowed models. Prompts and outputs aren't stored unless you turn on 7-day debug logging.

For this decision, what it doesn't do matters as much. Aight documents three endpoints (POST /v1/chat/completions, POST /v1/embeddings and GET /v1/models) and none of the gateway features in the routing section: no BYOK, no automatic routing, no gateway-side fallbacks or caching, no batch route. With one key, a fallback is a second model ID in a client loop like the one above. If you hold negotiated provider discounts, BYOK on another gateway or a self-hosted gateway keeps them, and if most of your spend is batch work or one provider, going direct costs less. Compare each model's price on the models page with its maker's before moving traffic.

What developers report

  • "managing that extra layer of uncertainty can cost more than it saves" (Manifest, on shutting down the LLM router in its gateway after four months and 7,000 cloud users, 31 Jul 2026, link)
  • "Routing belongs on the client side" (a Hacker News commenter on the Manifest post, 31 Jul 2026, link)
  • "You cant fail over to a different model though." (a Hacker News commenter, who added that switching providers isn't automatic either, citing Bedrock and Anthropic's tool search tool, 31 Jul 2026, link)
  • The team behind Weave's coding-agent router said routers that pick each request independently spend more on agents through cache misses, and that its cache-aware router cut its own spend by 40% (Show HN, 26 Jun 2026, link)
  • Of the 28 paid and 400 free routers tested, one drained ether from a private key the researchers had planted (arXiv paper, 9 Apr 2026, link)

FAQ

What is the difference between an API gateway and an AI gateway?

An API gateway manages traffic to your own services and counts requests. An AI gateway manages calls out to model providers and counts tokens, so its limits, metrics and caching work on tokens and prompts. Microsoft's AI gateway is an extension of the Azure API Management gateway, not a separate product.

Is OpenRouter an AI gateway?

Yes, of the hosted, pooled-key kind. Its pricing page lists 600+ models from 80+ providers behind one key and one balance, with provider routing, model fallbacks and an Auto Router. It charges 5.5% on card purchases, and BYOK is free on the first $25,000 of list-price usage a month.

Does an AI gateway add latency?

A little. A hosted gateway adds a network hop, and a self-hosted one adds processing time: LiteLLM's benchmark measured 2 ms median and 8 ms at p95 on four instances. Retries and fallbacks add more when they trigger, but they replace an error you would otherwise return.

What does BYOK mean in AI tools?

Bring your own key: the tool or gateway uses your own provider API key, so the provider bills you directly. Coding tools and gateways use the term the same way. In cloud security, BYOK means importing your own encryption key into a provider's key vault.

Do AI gateways store prompts?

It depends on the gateway and its settings. Cloudflare logs by default and its logs can include prompts, Vercel deletes prompt content when a request completes, and OpenRouter doesn't log prompts unless you opt in. The provider behind any gateway has its own retention policy too.


Sources

Checked on 5 November 2026. Updated 17 November 2026: Aight's current model list.