AI gateways in 2026: revisit LiteLLM, Bifrost and Agent Router before you put another model API key in application code

Most teams did not plan their first AI gateway. It arrived as a side effect. One service called a hosted model with a key from an environment variable. A second team copied the same pattern, with its own retry loop. A third team added a self-hosted model and wrote a small wrapper so that the code would not care which one it was talking to. A year later there are six keys in six places, three retry styles, no shared idea of what a request costs, and nobody can say which team spent how much on which model last month.

An AI gateway is the piece that sits between your applications and the model providers, whether those providers are hosted APIs or models you serve yourself. Applications talk to one endpoint with one kind of key. The gateway holds the real provider keys, decides where each request goes, retries or falls back when a provider fails, counts tokens and money, applies limits and budgets per team, and writes logs and traces in one format. In 2026 most of these gateways also act as a gateway for MCP tool servers, which brings its own security surface.

This post revisits three open-source choices for that job: LiteLLM, the Python proxy and SDK that many teams already have somewhere in their stack; Bifrost, a Go gateway from Maxim AI that pitches low overhead; and Agent Router, the project that was called Envoy AI Gateway until September 2026 and is built on Envoy Proxy and Envoy Gateway. The aim is not to crown a winner. The aim is to help you decide whether you need a gateway at all, which shape suits your team, and what to check before you trust any of them with every provider key you own.

If you are new to this series, a few earlier posts connect directly. The model serving side, where you run vLLM, SGLang or llama.cpp behind an OpenAI-compatible endpoint, is covered in LLM inference serving in 2026; this post starts where that one stops. General purpose gateways such as Kong, APISIX and Traefik are in API gateways in 2026. Where the provider keys themselves should live is in Vault and OpenBao and Kubernetes secrets delivery. Traces for model calls fit into the setup from distributed tracing in 2026. Retries that cost money are the subject of idempotency keys, and the supply chain lessons in this post connect with secret scanning and Python dependency tooling.

A scenario to keep us honest

The company in this section is invented. Any match with a real company is an accident.

Malabar Leaf Exports is a mid-sized spice and tea exporter in Kochi with about 220 staff and a technology team of 14. Over the last eighteen months the team has built four things that call language models:

• A customer support assistant on the website that answers order and shipment questions in English and Malayalam, using a hosted model.
• A nightly job that reads supplier invoices and packing lists and extracts line items, using a second hosted provider because it was cheaper for that job.
• A sales email drafting tool used by the export desk, built by a contractor, which calls whichever model the contractor liked at the time.
• An internal search assistant over quality certificates and lab reports, running on a small open-weight model served with vLLM on one GPU server in the office, because the documents should not leave the building.

Each of these has its own provider key in its own .env file or Kubernetes Secret. Two of them share one key. The contractor’s tool has a key that was once pasted into a public issue by mistake, rotated in a hurry, and pasted into three places again. The finance head asks every month why the model bill moved, and the honest answer is “we are not sure which app did it”. In March one provider had a two-hour outage and the support assistant showed an error page, even though the invoice job’s provider was healthy the whole time.

The team lead, Anitha, has a simple brief: one place for provider keys, one way to retry and fall back, a cost number per app and per team, a hard monthly budget for the contractor tool, traces that join the existing Jaeger setup, and no new system that needs a full-time person. The team runs a small Kubernetes cluster for the website and internal tools, and some jobs still run on plain VMs. Most developers write Python; two write Go.

We will come back to Malabar Leaf at the end with a worked plan.

The short version

• If you only have one or two apps and one provider, a shared client library with a configurable base_url, sane timeouts and one retry policy is often enough. Do not add a gateway to look modern.
• LiteLLM is the widest in provider coverage and the most familiar to Python teams. Its proxy gives virtual keys, budgets, spend tracking and fallbacks in the MIT part, but it needs Postgres for keys and spend, and several controls (SSO beyond five users, audit logs, secret manager integration, per-key guardrails and more) are in a separately licensed enterprise directory. It also had two serious PyPI supply chain incidents in 2026, in March and in October, so pinning and provenance checks are not optional.
• Bifrost is a single Go binary or container with a web UI, Apache 2.0 licensed, with virtual keys, budgets, token and request limits, fallbacks, semantic caching and MCP in the open-source build. Its vendor claims very low added latency. It is younger and backed by one company, and several useful pieces (clustering, Vault support, guardrails, audit logs) are enterprise only.
• Agent Router (formerly Envoy AI Gateway) is the choice when you already run Envoy Gateway or want the gateway expressed as Kubernetes resources. It is Apache 2.0, now under the Agentic AI Foundation, and does token-based rate limiting through Envoy’s global rate limiter. It is the least “batteries included” for cost dashboards and budgets per team; you build those from metrics and policies.
• Whatever you pick, treat the gateway as the most sensitive service you run. It holds every provider key, often a database of virtual keys, and in 2026 all three projects shipped fixes for high or critical issues, many of them in MCP features.

Where things stand on 31 October 2026

These are the facts I could confirm from primary sources (project repositories, release pages, package indexes, security advisories and official blogs) on 31 October 2026. Times are converted to IST.

LiteLLM:

• The latest stable release on PyPI is 1.106.6, uploaded on 29 October 2026 at about 4:18 AM IST. Release candidates for 1.108 and dev builds for 1.109 were also on PyPI.
• The project ships a new stable minor line roughly every week and, since June 2026, gives patches to the four most recent stable minor lines. There is no long term support line. At the end of October that meant 1.103 to 1.106.
• The repository is MIT licensed except for the enterprise/ directory, which is under the BerriAI Enterprise licence. Production use of the enterprise features needs a paid licence key.
• The package now requires Python 3.10 or newer, below 3.15.
• On 21 October 2026 a malicious litellm 1.107.0 was published to PyPI between about 5:09 PM and 6:53 PM IST, and malicious Docker image tags were live for longer. More on this in the security section.

Bifrost:

• The latest release is transports v2.4.0, published on 30 October 2026 at about 1:35 AM IST. It added “relays” for non-LLM APIs, more providers and MCP prompts and resources.
• The 2.0 line arrived on 26 August 2026 with breaking changes and database migrations that cannot be reversed.
• The repository is Apache 2.0. An enterprise edition with extra features is sold by Maxim AI.

Agent Router:

• The latest stable release is v1.2.0 from early October 2026, the first release under the new name. A v1.2.1 release candidate (rc2) appeared on 28 October.
• The project reached 1.0 in June 2026 with a commitment to compatibility across 1.x.
• On 10 September 2026 Envoy AI Gateway was renamed Agent Router and joined the Agentic AI Foundation (AAIF), which also hosts MCP and A2A. The Kubernetes API group, CRD names, container images and Helm chart locations did not change, so existing manifests keep working.
• The repository is Apache 2.0.

A note on a name you may expect here. Portkey’s open-source gateway was a strong candidate a year ago. In 2026 Palo Alto Networks acquired Portkey (announced 30 April, completed 29 May), the product is now sold as Prisma AIRS AI Gateway, the last tagged open-source release I could find is v1.15.2 from January 2026, and the announced “Gateway 2.0” open-source merge was never tagged as a release. The MIT code is still on GitHub and still works, but I would not start a new deployment on it without a clear answer about its future. That is why it is not one of the three here.

The first question: do you need a gateway, or just a shared client?

Before comparing tools, be honest about which problem you have. There are three different problems that people solve with an “AI gateway”:

1. Code problem: every app has its own HTTP client, timeouts and retry loop for model calls. This is solved by a shared internal library. Most providers and most self-hosted servers speak the OpenAI-compatible API, so a thin wrapper around the official SDK with a configurable base_url covers a lot.
2. Control problem: you need one place for provider keys, budgets and limits per team or app, and an audit of who called what. This needs a service, because a library cannot stop a team from exceeding a budget or keep the real key away from the app.
3. Platform problem: you run many models, some hosted, some self-hosted, across regions or clusters, and you want routing, traffic splitting and token-based limits as part of your existing network layer. This is where a gateway built on a proxy such as Envoy fits.

If you only have the first problem, solve it with a library and stop there. If you have the second, any of the three tools can help, and the choice is mostly about language, operations and licence. If you have the third, Agent Router or the AI features of your existing API gateway deserve the first look.

A useful test: ask “what do we lose if the gateway is down for ten minutes?” If the answer is “every AI feature in the company”, you are taking on a new tier one service. Plan for that, or keep the library approach until you are ready.

What each tool is

LiteLLM

LiteLLM started in mid 2023 as a Python SDK that let you call many providers with one OpenAI-style function. It grew a proxy server (“LiteLLM Proxy” or “AI Gateway”) that exposes an OpenAI-compatible HTTP API, plus an admin UI. The SDK and proxy share the same core: provider adapters for more than 100 providers, a router with load balancing, retries and fallbacks, cost calculation from a model price map, and logging callbacks for many observability tools.

The proxy reads a YAML config. Provider keys can be pulled from environment variables. Virtual keys, teams, users, budgets and spend logs live in Postgres; without a database you get routing, retries, fallbacks and caching, but not virtual keys or spend tracking. Redis is used for shared rate limit counters and caching when you run more than one proxy instance.

A small proxy config looks like this (example):

# config.yaml (example)
model_list:
  - model_name: support-chat
    litellm_params:
      model: openai/<hosted-model>
      api_key: os.environ/PROVIDER_A_KEY
  - model_name: support-chat-backup
    litellm_params:
      model: anthropic/<hosted-model>
      api_key: os.environ/PROVIDER_B_KEY
  - model_name: docs-search
    litellm_params:
      model: openai/<local-model>
      api_base: http://vllm.internal:8000/v1
      api_key: os.environ/VLLM_KEY
router_settings:
  num_retries: 2
  fallbacks: [{"support-chat": ["support-chat-backup"]}]
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

A team then gets a virtual key with its own budget and limits (example):

# create a virtual key with a monthly budget (example)
curl -s http://litellm.internal:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"models": ["support-chat"], "max_budget": 200, "budget_duration": "30d", "rpm_limit": 60, "tpm_limit": 100000, "metadata": {"team": "support"}}'

The application keeps using the OpenAI SDK, pointed at the proxy, with the virtual key in place of a provider key.

What is open and what is not matters here. In the MIT part you get the proxy, virtual keys, budgets and rate limits, spend tracking, the admin UI, load balancing, the guardrails framework with custom guardrails and Presidio for PII masking, and the MCP gateway. In the enterprise directory, under a separate licence, are SSO for more than five users, SCIM, JWT auth, audit logs, role based access with organisations, IP allowlists, key rotation, secret manager integrations (AWS, Azure, Google, HashiCorp Vault, CyberArk), team based logging, tag budgets, model-specific budgets per key, soft budget alerts, per-key and per-team guardrails, and several ready made guardrail integrations. The enterprise build is the same image with a licence key. BerriAI’s documentation is clear about this split; read it before you design your access model around a feature.

Bifrost

Bifrost is a gateway written in Go by Maxim AI, first published in March 2025. You can start it with npx -y @maximhq/bifrost or the maximhq/bifrost container image; it listens on port 8080 and serves a web UI for configuration. It also has a Go SDK if you want to embed the core in a Go service. Applications point their existing SDKs at a provider-shaped path, for example http://bifrost.internal:8080/openai, so the change in code is usually just the base URL and key.

The open-source build includes a unified OpenAI-compatible API, fallbacks across providers, load balancing across multiple keys for the same provider, a semantic caching plugin, an MCP gateway, Prometheus metrics, OpenTelemetry export and request logging. Governance is built around virtual keys (they start with sk-bf-), which can carry budgets with reset periods from one minute to one year, token and request rate limits, an allow list of providers and models, expiry, and a hierarchy of teams and customers. Errors are explicit: 402 when a budget is exhausted, 429 when a token or request limit is hit, 403 when a model is not allowed for that key.

One default deserves attention. Governance can be optional: unless “enforce auth on inference” is on, requests without a virtual key may still be served. In the 2.x line new deployments have it on by default, but an upgraded deployment keeps whatever value it had stored. Check this setting on every Bifrost you run.

A provider block in config.json looks roughly like this (example; check field names against the docs for your version):

{
  "providers": {
    "openai": {
      "keys": [
        { "name": "provider-a-main", "value": "env.PROVIDER_A_KEY", "models": ["<hosted-model>"], "weight": 1.0 }
      ]
    },
    "anthropic": {
      "keys": [
        { "name": "provider-b-main", "value": "env.PROVIDER_B_KEY", "models": ["<hosted-model>"], "weight": 1.0 }
      ]
    }
  }
}

The enterprise edition, with a 14-day trial, adds SAML SSO and role based access, adaptive load balancing, cluster mode, alerts, log exports, audit logs, Vault integrations (HashiCorp, AWS, Google, Azure), VPC deployment, guardrails, custom plugins and “Projects”.

Agent Router (formerly Envoy AI Gateway)

Agent Router began in October 2024 when Bloomberg brought the problem of governing model traffic to the Envoy community, and Tetrate and Bloomberg built the first version together. It is not a new proxy. It is a control plane and an external processor that extend Envoy Gateway, which in turn implements the Kubernetes Gateway API on Envoy Proxy. Requests flow through Envoy; an external processing service understands model request bodies, picks the backend, translates between API shapes, injects provider credentials and extracts token usage for rate limiting and metrics.

You describe the setup with Kubernetes resources. An AIGatewayRoute matches requests (usually on the model name, which the gateway exposes as the x-ai-eg-model header), lists backends with priorities for fallback, and declares which token counts should be recorded. An AIServiceBackend describes a provider or a self-hosted server and its API schema. A BackendSecurityPolicy says where the provider credential comes from: an API key in a Secret, or cloud credentials for AWS, Azure or GCP. There is also an aigw CLI that can run the gateway as a standalone process on Linux or macOS without Kubernetes; the documentation calls the CLI experimental.

A minimal route looks roughly like this (example; check the API version and fields against the docs for your release):

# AIGatewayRoute with fallback and token cost recording (example)
apiVersion: aigateway.envoyproxy.io/v1beta1
kind: AIGatewayRoute
metadata:
  name: support-chat
  namespace: ai-gw
spec:
  parentRefs:
    - name: ai-gateway
      kind: Gateway
      group: gateway.networking.k8s.io
  rules:
    - matches:
        - headers:
            - type: Exact
              name: x-ai-eg-model
              value: support-chat
      backendRefs:
        - name: provider-a
          priority: 0
        - name: provider-b
          priority: 1
  llmRequestCosts:
    - metadataKey: llm_total_token
      type: TotalToken

Token limits per team are then set with an Envoy Gateway BackendTrafficPolicy that uses global rate limiting (backed by Redis), keyed on a tenant header and the model header, with the cost taken from the llm_total_token metadata. There is no built-in money budget or spend dashboard in the way LiteLLM and Bifrost have one; you get metrics that follow the OpenTelemetry GenAI conventions and build the reports yourself.

Side by side

Licence and governance

LiteLLM is MIT with an enterprise directory under a commercial licence. It is run by one company, BerriAI. Development is very fast, with a stable minor release almost every week. That speed is good for new providers and bad for anyone who likes to read release notes before upgrading.

Bifrost is Apache 2.0 for the open-source build and is run by Maxim AI (legal name H3 Labs Inc.). The enterprise edition is a separate commercial product. It is the youngest of the three and has the smallest contributor base outside the vendor.

Agent Router is Apache 2.0 and, since September 2026, a project of the Agentic AI Foundation. The project reports 132 contributors from 21 organisations and 11 public adopters including Bloomberg, Tencent Cloud, Nutanix and LY Corporation; those numbers are the project’s own. It has no enterprise directory in the repository; commercial support comes from vendors such as Tetrate.

If vendor neutrality matters to your organisation, Agent Router is the clear choice. If you are fine buying from a single vendor later, read the enterprise feature lists of LiteLLM and Bifrost now, so you know which line you will cross and when.

Deployment model

• LiteLLM: Python service, usually the official container, plus Postgres for keys and spend, plus Redis when you run more than one instance. The admin UI is part of the proxy.
• Bifrost: one Go binary or container, config in a file or in its own store through the UI, web UI included. Cluster mode across instances is an enterprise feature.
• Agent Router: Envoy Gateway and the Agent Router controller in Kubernetes, Envoy data plane pods, Redis for global rate limiting. Or the experimental standalone CLI for local use.

For a small team without Kubernetes, LiteLLM or Bifrost on two VMs behind a load balancer is easier to reason about. For a team that already runs Envoy Gateway for north-south traffic, Agent Router adds model awareness to something you already operate.

Providers and routing

LiteLLM has the longest provider list and usually adds new models and providers first. Its router supports several load balancing strategies, cooldowns for failing deployments, retries with backoff, fallbacks between model groups and special fallbacks for context window or content policy errors.

Bifrost covers the major hosted providers and OpenAI-compatible servers, and keeps adding more (v2.4.0 added MiniMax, BytePlus and SageMaker). It load balances across keys by weight and supports fallbacks in config or per request. Adaptive load balancing based on live performance is an enterprise feature.

Agent Router supports a shorter list of providers, such as OpenAI, Anthropic, AWS Bedrock, Azure OpenAI and GCP Vertex AI, plus OpenAI-compatible self-hosted servers, and translates between their API shapes. Routing uses Envoy’s machinery: priorities for fallback, weights for traffic splitting, header matches, and the Gateway API Inference Extension for routing to self-hosted model pods based on their load. That last point matters if you serve models yourself on Kubernetes; it links naturally with the serving post mentioned above.

Keys, budgets and rate limits

This is the core reason most teams want a gateway, and the three differ most here.

LiteLLM: virtual keys with max budget, budget reset period, requests per minute, tokens per minute, max parallel requests and allowed models, at key, user, team and (enterprise) organisation level. Spend is calculated per request from a price map and stored in Postgres. Soft budget alerts, tag budgets and model-specific budgets per key are enterprise.

Bifrost: virtual keys with budgets (with reset durations), token and request limits, allowed providers and models, expiry, and a teams and customers hierarchy, all in the open-source build. Clear error codes help client code tell “out of budget” apart from “slow down”.

Agent Router: token-based rate limits through Envoy Gateway global rate limiting, keyed on any header such as a tenant id, using input, cached input, output or total tokens, or a CEL expression that weights them. Upstream quotas per backend are expressed with a QuotaPolicy. Two details from the documentation are worth knowing. Usage is charged after the response completes, and a streaming response is not cut off when the limit is crossed, so a tenant can overshoot by one large response. And the limiter needs Redis.

Whichever you choose, remember that budgets are only as good as the price data behind them. LiteLLM and Bifrost calculate cost from a price table; a self-hosted model has no price unless you give it one; and a provider’s price change does not reach your gateway until the table is updated.

Caching

LiteLLM supports exact-match caching in memory, Redis, S3 and others, and semantic caching with Redis or Qdrant. Bifrost has a semantic caching plugin that needs a vector store and an embedding model. Agent Router does not focus on response caching; it does record cached input tokens reported by providers, which matters for cost.

Treat caching of model responses with care. Exact-match caching is safe for deterministic jobs such as classification with temperature zero. Semantic caching returns an answer to a “similar” question, which can be wrong in a support setting where order numbers and dates matter. If you use it, scope the cache per tenant and per feature, and measure how often it returns something a human would reject. The vector side of this is the same trade-off discussed in vector search in 2026.

Guardrails

LiteLLM has a guardrails framework in the open-source part, with custom guardrails and Presidio for PII masking, and many ready integrations, several of them enterprise only. Per-key and per-team guardrails are enterprise. Bifrost lists guardrails as an enterprise feature. Agent Router leaves content checks to external services that you call through Envoy filters or the external processor.

A practical view: a gateway is a good place for cheap, generic checks such as blocking requests with obvious secrets or masking PII before it leaves your network. It is a poor place for business rules that only one app understands. Keep those in the app.

Observability

All three export metrics and traces, and all three move towards the OpenTelemetry GenAI semantic conventions (gen_ai.* attributes for model, token usage and so on). LiteLLM also has callbacks for many LLM observability products and writes spend logs to Postgres. Bifrost has Prometheus metrics, OpenTelemetry export with a separate metrics pipeline since 2.0, request logs in its UI, and header redaction before export. Agent Router emits Envoy’s usual metrics and access logs plus GenAI metrics; v1.2.0 added a gen_ai.backend attribute, which may change your dashboards.

Decide early whether prompts and responses go into logs at all. They often contain customer data. Bifrost lets you turn content logging on or off per virtual key; LiteLLM has settings to avoid storing message content. Whatever you choose, apply the same retention and access rules you use for application logs, as discussed in the log pipeline post.

Performance

Here, vendor numbers are loud and independent numbers are rare.

Bifrost’s README claims about 11 microseconds of added overhead at a sustained 5,000 requests per second on an AWS t3.xlarge instance (59 microseconds on a t3.medium), with 100 percent success at that rate. That is the vendor’s own benchmark. It is plausible for a Go proxy doing little work per request, and it will rise as you turn on logging, governance with a database, caching and plugins.

LiteLLM is a Python service and is usually run with several workers and instances when throughput is high. For most teams the model call takes hundreds of milliseconds to many seconds, so a few milliseconds in the gateway is not the bottleneck. It does matter for high volume, short calls such as embeddings or classification.

Agent Router inherits Envoy’s data plane performance, with the external processor adding a hop for body processing. I did not find an independent benchmark I trust for 2026 versions of any of the three. Measure on your own traffic: run the same request mix through no gateway and through the gateway, with your real logging settings on, and compare p50 and p99.

Operations and upgrades

LiteLLM’s weekly stable lines and four-line patch window mean you either upgrade every few weeks or fall out of support within a month. Plan for it like a dependency that moves fast, with automated update pull requests and a staging proxy, in the spirit of the dependency updates post. Database migrations run with upgrades; back up Postgres first.

Bifrost’s 2.0 upgrade included database migrations that cannot be reversed, and some breaking changes. Take a backup and test the upgrade on a copy before you touch production. Since then, minor releases have come every few weeks.

Agent Router follows a 1.x compatibility promise, but v1.2.0 still came with security-driven behaviour changes: the Helm chart no longer ships a default MCP session seed, MCP OAuth tokens must match the issuer, CEL evaluation errors now deny instead of allow, cross-namespace Secret references need a ReferenceGrant, and per-tenant token limits that were silently ignored are now enforced. The upgrade notes say to upgrade Envoy Gateway first and to come to 1.2 from 1.1, not from 1.0. Read release notes for every minor version.

Security

This is the section to read twice. A gateway holds every provider key, often a database of virtual keys, and sometimes the ability to start MCP tool processes. In 2026 all three projects published serious advisories, and LiteLLM had two supply chain attacks.

LiteLLM supply chain. On 24 March 2026 two malicious versions, 1.82.7 and 1.82.8, were on PyPI for about 40 minutes from about 4:09 PM IST. They contained a credential stealer; 1.82.8 used a .pth file so that it ran whenever Python started. The project linked the cause to a compromised security scanner in its CI pipeline, the official Docker image was not affected, and the PyPI blog later estimated about 119,000 downloads in that window, with a large share of installs unpinned. LiteLLM rebuilt its release pipeline and published 1.83.0 from it.

On 21 October 2026 it happened again. A malicious 1.107.0 was on PyPI from about 5:09 PM to 6:53 PM IST (all wheels), and malicious Docker Hub tags (v1.107.0, latest, main-latest, main-stable) were live from about 5:16 PM to 7:50 PM IST, with GHCR affected until about 7:21 PM IST. The payload again used startup hooks (.pth and sitecustomize.py) to collect credentials and load a remote access tool. The source tag on GitHub was clean, and the enterprise private registry was not affected. The project’s postmortem of 23 October traces it to an admin key for an internal LiteLLM platform that was committed to a public repository in May, deleted three hours later but never revoked. In October attackers used that key to take over the deployment, extract an unscoped GitHub CLI token of an organisation owner, read CI secrets, override branch protection and run a release. The advice was to pin to a known good version, remove the malicious files, rebuild images and rotate everything the proxy could reach: provider keys stored in the proxy database, master and virtual keys, and the database URL.

There are two lessons, and neither is “never use LiteLLM”. First, a gateway’s package and image are part of your secret perimeter: pin exact versions, use hashes or digests, prefer delayed adoption of new releases (a few days is enough to catch most of these), and scan installed packages for unexpected .pth and sitecustomize.py files. Second, a deleted secret is not a revoked secret. That is exactly the case secret scanning exists for, and it applies to your own repositories too. Image signing and attestations, discussed in container image signing, help only if you verify them at deploy time.

LiteLLM advisories. The GitHub advisory list for 2026 is long. A few that shape deployment choices: a critical SQL injection in API key verification (CVE-2026-42208, fixed in 1.83.7), a critical authentication bypass via the Host header (CVE-2026-49468, fixed in 1.84.0), a high severity MCP authentication bypass with OAuth2 passthrough (CVE-2026-59822, fixed in 1.84.0), command execution through MCP stdio test endpoints (CVE-2026-42271, fixed in 1.83.7), a sandbox escape in custom-code guardrails (CVE-2026-40217), a critical privilege escalation through salt key reuse (CVE-2026-114612, fixed in the 1.100 to 1.103 patch lines), and on 27 October a critical unauthenticated file read that can leak the master key on proxies running with an enterprise licence (GHSA-w4c7-34wj-hqm8, fixed in 1.103.10, 1.104.8, 1.105.6 and 1.106.5). Mitigations recommended in that last advisory are worth applying anyway: keep the admin surface off the internet, restrict egress, set a separate LITELLM_SALT_KEY, and keep provider keys in a secret manager.

Bifrost advisories. On 23 September 2026 two advisories were published: a critical remote code execution through MCP stdio clients registered via the management API when auth was disabled (the default at the time), and a high severity unauthenticated code execution through the custom plugin path on dynamically linked builds. Both are fixed in 2.1.0. An earlier SSRF deny-list gap was fixed in core 1.5.16. Since 2.0 Bifrost asks for a setup token on first run, and 2.1 stopped unauthenticated callers from registering stdio MCP clients.

Agent Router advisories. Around the v1.2.0 release (late September to early October 2026) the project published about ten advisories, mostly in MCP handling: CEL errors failing open in MCP authorisation, MCP session resumption without verifying the subject (with a forgeable default Helm seed), unbounded MCP request bodies, MCP authorisation without JWT validation, BackendSecurityPolicy reading Secrets from other namespaces, and a duplicated stream_options field that suppressed token accounting. The release notes for v1.2.0 list the fixes.

The common thread: MCP support turned AI gateways from “HTTP proxy with keys” into “service that can start processes and call tools”. If you do not use the MCP features, turn them off or block those paths. If you do, treat the gateway like a CI runner, not like a reverse proxy.

When a thin client library is still fine

A gateway is not the only good answer. A small shared library is fine, and often better, when:

• You have one or two apps and one provider, and the bill is small enough that per-team reporting does not matter yet.
• Keys already come from a secret manager at runtime and nobody copies them into .env files.
• You can afford the app failing when the provider fails, or your fallback is “show a polite message”.
• Your platform team is two people and cannot carry another tier one service.

In that case, write the library well. Use the official SDK with a configurable base_url so you can add a gateway later without changing call sites. Set explicit timeouts. Retry only on 429 and 5xx with backoff and jitter, and never blindly retry a call that triggers side effects, as the idempotency post argues. Record model, token counts and latency in your existing traces. Put a feature flag around the model choice, using the approach in feature flags in 2026, so switching providers is a config change. When the second or third team arrives, you will know which gateway features you actually need.

A worked plan for Kochi

Back to Malabar Leaf. The team writes mostly Python, runs a small Kubernetes cluster but no Envoy Gateway, and has one GPU server running vLLM. They want budgets and per-team cost without building dashboards from scratch. That points to LiteLLM or Bifrost. Both are reasonable. Anitha picks Bifrost for the trial because it is a single container with fewer moving parts, then keeps LiteLLM as the fallback option if a provider they need is missing. A team with more Python habits or a need for a provider only LiteLLM supports could make the opposite choice with the same plan.

1. Week 1: inventory. List every place a model key exists: .env files, Kubernetes Secrets, CI variables, laptops, the contractor’s repository. Run a secret scanner on all repositories, including history. List each app, its provider, its model and rough monthly spend from provider invoices.
2. Week 1: decide what the gateway will not do. No semantic caching for the support assistant. No MCP features in phase one; block the MCP paths at the ingress. Guardrails stay in the apps.
3. Week 2: deploy two Bifrost instances in the cluster in their own namespace, with a pinned image digest, behind an internal-only Service. No public ingress. Admin UI reachable only from the office VPN. Confirm “enforce auth on inference” is on. Provider keys come from Kubernetes Secrets synced from the company’s secret manager, not typed into the UI.
4. Week 2: restrict egress from the gateway namespace to the provider endpoints and the vLLM server only. This limits the damage if the gateway or one of its plugins is ever compromised.
5. Week 3: create one virtual key per app, not per developer. Give the contractor tool a hard monthly budget and a token limit, and allow only one model. Give the support assistant a higher limit and a fallback from provider A to provider B.
6. Week 3: wire OpenTelemetry export into the existing collector and Jaeger. Turn off content logging for the support assistant key; keep it on, with short retention, for the invoice job while they debug extraction quality.
7. Week 4: move the invoice job first, because it is batch and easy to roll back. Change only the base URL and key. Compare token counts and cost reported by the gateway with the provider’s console for a week.
8. Week 5: move the support assistant behind a feature flag, 10 percent of traffic first. Test fallback by blocking provider A from the gateway namespace for five minutes in staging.
9. Week 6: move the contractor tool and the internal search assistant. Rotate every old provider key once all apps are moved, and delete the old Secrets.
10. Ongoing: an update pull request for the gateway image every two weeks, applied to staging first, with release notes read by a person. Subscribe to the project’s security advisories. Review spend per virtual key monthly with finance.

What they deliberately did not do: they did not put the gateway on the internet, did not let the gateway start MCP processes, did not let developers use the master key, and did not promise finance that gateway cost numbers would match invoices to the rupee. Price tables lag; they agreed on “within a few percent, reconciled monthly”.

When each is better

LiteLLM is better when:

• You need the widest provider coverage or a provider others do not support yet.
• Your team is comfortable with Python and wants to use the same router logic in-process (SDK) and as a service (proxy).
• You want budgets, spend logs and an admin UI in the open-source part and can run Postgres.
• You can keep up with a weekly release train.

Bifrost is better when:

• You want a single binary or container with a UI and low overhead, and no Python runtime in the path.
• You want virtual keys, budgets, token limits and semantic caching in the open-source build.
• Your team writes Go and might embed the core.
• You are fine with one vendor behind the project.

Agent Router is better when:

• You already run Envoy Gateway or plan to standardise on the Gateway API.
• You want the gateway in the same GitOps flow as the rest of your Kubernetes resources.
• You serve your own models on Kubernetes and want inference-aware routing to model pods.
• Vendor-neutral governance under a foundation matters to you.

When not to adopt each

Do not adopt LiteLLM if you cannot pin, verify and upgrade a fast-moving Python dependency with discipline, if you need SSO and audit logs but cannot buy the enterprise licence, or if you plan to expose the admin endpoints to the internet.

Do not adopt Bifrost if you need clustering, Vault integration or audit logs without paying, if your organisation needs a project with many independent maintainers, or if you are not ready to check its auth settings after every upgrade.

Do not adopt Agent Router if you do not run Kubernetes and do not want to (the standalone CLI is experimental), if you need a ready spend dashboard and money budgets per team on day one, or if nobody on the team understands Envoy and the Gateway API well enough to debug a route.

And do not adopt any gateway if the only problem you have is duplicated client code.

A short history

The pattern is older than the name. Teams put API gateways in front of payment providers and SMS providers for the same reasons: one key, one retry policy, one bill per team. When hosted model APIs became common in 2023, the first answers were libraries. LiteLLM started in July 2023 as an SDK to call many providers with one function, and its proxy followed as teams asked for keys and budgets in one place.

In 2024 the space filled up. Portkey released an open-source gateway in TypeScript. API gateway vendors added AI plugins, which we touched on in the API gateways post. In October 2024 Bloomberg and Tetrate brought the problem to the Envoy community, and Envoy AI Gateway was announced, with its first release in February 2025. Maxim AI published Bifrost in March 2025 with a focus on overhead.

2026 was a year of consolidation and hard lessons. Envoy AI Gateway reached 1.0 in June and, in September, became Agent Router under the Agentic AI Foundation. Portkey was acquired by Palo Alto Networks and its product folded into Prisma AIRS. Bifrost shipped 2.0 with governance and MCP at the centre. LiteLLM grew fast and was hit twice by supply chain attacks. And MCP support, added by everyone, produced a large share of the year’s security advisories across all three projects.

Decision guide

Your team has one app and one provider

Write a shared client library. Pull keys from a secret manager. Revisit in six months.

You need budgets and per-team cost, and you do not run Envoy Gateway

Trial LiteLLM and Bifrost side by side for a week each with the same two apps. Choose on provider coverage, how your team feels operating Postgres (LiteLLM) versus a single service (Bifrost), and which enterprise features you would need within a year.

You already run Envoy Gateway or Kubernetes Gateway API

Start with Agent Router. Add token-based limits per tenant and export GenAI metrics to your existing stack. Build cost reports from those metrics.

You serve your own models on Kubernetes

Agent Router with the Gateway API Inference Extension is the natural fit for routing to model pods. You can still put LiteLLM or Bifrost in front for budgets if you need them, but avoid three layers of proxies without a clear reason.

You already run Kong or APISIX

Look at their AI plugins first. A second gateway product is a cost you should justify.

You are on Portkey’s open-source gateway today

It still runs, and nothing forces you to move this month. Pin the version, block the admin routes from outside, and plan a migration or a commercial decision within the next two quarters.

Checklist

1. Inventory every model key and the apps using it.
2. Decide which of the three problems (code, control, platform) you are solving.
3. Pin the gateway version and image digest; never use latest.
4. Delay new releases by a few days unless they fix a security issue that affects you.
5. Scan the installed package or image for unexpected startup hooks.
6. Keep the gateway and its admin UI off the internet.
7. Restrict egress from the gateway to provider endpoints and your model servers.
8. Store provider keys in a secret manager; never in the gateway’s UI fields if you can avoid it.
9. Enforce authentication on inference routes; check after every upgrade.
10. Issue one virtual key per app or team, not per developer, and never share the master key.
11. Set budgets and token limits on every key, even generous ones.
12. Decide what gets logged: metadata always, content only where needed, with retention.
13. Export traces and metrics to your existing observability stack.
14. Test fallback by actually breaking a provider in staging.
15. Disable MCP features you do not use, or block their paths.
16. Subscribe to the project’s security advisories and keep a rotation runbook ready.

Common mistakes

• Treating the gateway like a stateless reverse proxy. It is a secret store with an HTTP interface. Protect it like one.
• Running the admin UI on the same public hostname as the inference API.
• Retrying non-idempotent calls. A model call that triggers a tool to send an email or create an order should not be retried blindly by the gateway.
• Trusting cost numbers as invoices. They are estimates from price tables.
• Turning on semantic caching for user-specific questions and returning someone else’s answer.
• Upgrading a fast-moving gateway straight to production on release day. Both LiteLLM incidents were caught within hours; teams that waited a day were safe.
• Deleting a leaked key from a repository and not revoking it.
• Assuming “open source” means every feature is free. Read the enterprise lists before you design around a feature.
• Building three layers of proxies (API gateway, AI gateway, serving router) without knowing which one owns retries. Retries at every layer multiply.

The drill we ran

I ran a small, local drill on 31 October 2026 at about 2:30 AM IST, to see LiteLLM’s retry, fallback and cache behaviour with my own eyes. Everything ran on loopback with fake keys; no real provider was called.

• I installed litellm[proxy]==1.106.6 in a virtual environment with uv. This version came after both incidents and is not one of the known malicious versions. PyPI’s provenance endpoint returned no attestation for the 1.106 wheels I checked, so I could not verify provenance that way. Before starting Python, I checked site-packages: there were no .pth files and no sitecustomize.py.
• A small Python stub on 127.0.0.1 pretended to be an OpenAI-compatible provider: one model always returned 500, one always returned 429, one returned a normal reply.
• The proxy config had three model groups, num_retries: 2, fallbacks from the failing groups to the healthy one, an in-memory cache, and a master key. No database.

What I saw:

• A request to the group that always returned 500 reached the stub three times (one try plus two retries), then fell back to the healthy model and returned 200 to the client. The response headers said x-litellm-attempted-fallbacks: 1 and named the model group that answered. The whole call took about 6 seconds, almost all of it retry backoff. In a user-facing app, that delay matters as much as the fallback itself.
• The 429 group behaved the same way: three attempts, then fallback, about 4 seconds.
• With the cache on, two identical requests reached the stub only once; the second came back with a cache key header.
• A request without a key was rejected and did not reach the stub, but the status code was 500, not 401, because the error handler tried to load the database client that I had not installed. Creating a virtual key with a budget failed with “No connected db”. So virtual keys, budgets and spend tracking were not tested; they need Postgres.
• Cost headers showed zero, because my fake model names were not in the price table. That is a small reminder of how budgets depend on price data.

Not run: Bifrost, Agent Router, Postgres, Redis, budgets, real providers and any load or latency test. I stopped both processes at the end and confirmed with pgrep that nothing was left running.

Unlearn, re-learn

Unlearn that a model API key is just another config value for each app. It is spend authority and data access in one string, and it belongs in one controlled place.

Unlearn that a gateway is “just a proxy”. In 2026 an AI gateway can hold every key, run tools and start processes. Its security advisories read like those of a CI system.

Re-learn the old API gateway lessons: one key per consumer, limits on every consumer, retries owned by one layer, logs with clear retention.

Re-learn supply chain hygiene for your infrastructure software, not only for your app dependencies. Pin, verify, wait a little, and revoke what leaks.

Practise it small: one app, one virtual key, one budget, one broken provider in staging. Then decide.

Sources

LiteLLM:

• GitHub repository and LICENSE (MIT except enterprise/): https://github.com/BerriAI/litellm
• PyPI release history and package metadata (1.106.6, Python requirement): https://pypi.org/project/litellm/
• Documentation, proxy, virtual keys, budgets, enterprise features: https://docs.litellm.ai/
• Release cycle and version support policy (docs, release notes)
• Security advisories: https://github.com/BerriAI/litellm/security/advisories
• March 2026 incident: LiteLLM security update, PyPI advisory PYSEC-2026-2, PyPI blog of 2 April 2026
• October 2026 incident: LiteLLM security notice for v1.107.0 and postmortem “The attack chain behind the v1.107.0 LiteLLM release” (23 October 2026), Microsoft Security blog (23 October 2026)

Bifrost:

• GitHub repository, README (features, vendor benchmark) and LICENSE (Apache 2.0): https://github.com/maximhq/bifrost
• Releases, including transports v2.4.0, v2.1.0 and v2.0.0: https://github.com/maximhq/bifrost/releases
• Documentation, governance, virtual keys, enterprise features: https://docs.getbifrost.ai/
• Security advisories: https://github.com/maximhq/bifrost/security/advisories

Agent Router (formerly Envoy AI Gateway):

• GitHub repository and releases (v1.2.0, v1.2.1-rc2): https://github.com/theagentrouter/agent-router
• Documentation, CLI, token rate limiting, upgrade notes: https://theagentrouter.ai/docs/
• Tetrate blog on the rename and move to the Agentic AI Foundation (September 2026)
• Security advisories: https://github.com/theagentrouter/agent-router/security/advisories

Portkey:

• GitHub repository and releases: https://github.com/Portkey-AI/gateway
• Palo Alto Networks announcements on the Portkey acquisition and Prisma AIRS AI Gateway (2026)

Related posts on Revisit.Tech: LLM inference serving in 2026, API gateways in 2026, Vault and OpenBao, Kubernetes secrets delivery, distributed tracing in 2026, idempotency keys, feature flags in 2026, Python dependency tooling in 2026, secret scanning in 2026, container image signing in 2026, dependency updates in 2026, vector search in 2026, log pipeline.