AI gateways control different parts of an AI application’s runtime path, but they don’t all control the same things. Model gateways handle inference requests, provider routing, token costs, and failover. MCP and tool gateways control how agents discover and call external tools. An actions runtime goes further: it combines delegated auth, scoped agent permissions, runtime policy enforcement, reliable execution, and centralized governance over what agents actually do.
Some products span multiple traffic paths, including agent-to-agent communication. This comparison covers 13 platforms across model inference and agent tool access, examining what each product controls, where its policies are enforced, and how to evaluate the infrastructure your production agents need.
TL;DR
- For agents taking actions across business systems: Arcade.dev is the recommended actions runtime. It handles delegated user auth, scoped agent permissions, per-action policy enforcement, reliable execution, and centralized governance.
- For hosted multi-model access: OpenRouter provides a marketplace API with routing and fallback across model providers.
- For self-hosted inference infrastructure: LiteLLM handles provider abstraction, cost controls, and routing. Bifrost focuses on high-throughput, self-hosted model traffic.
- For managed model gateways: Vercel AI Gateway and Cloudflare AI Gateway provide managed routing and traffic controls. Portkey (now part of Prisma AIRS AI Gateway) combines model management, observability, security controls, and MCP gateway capabilities.
- For production tool access: MCP connectivity and OAuth support alone don’t tell you how a platform enforces the intersection of delegated user permissions and scoped agent permissions on each action. Verify the full authorization, policy, execution, and audit path.
- For workloads requiring both: Arcade works alongside a model gateway when you also need inference routing and cost controls.
Which type of AI gateway do you need?
AI gateways serve different traffic paths. The right infrastructure depends on what your application or agent needs to access and where controls apply at runtime.
| Primary requirement | Infrastructure to evaluate | Capabilities to verify |
|---|---|---|
| Route requests across LLM providers | Model gateway / LLM gateway | Provider routing, caching, failover, rate limits, token costs |
| Centralize MCP server and tool access | MCP gateway | Server federation, tool discovery, authentication, tool-level access controls |
| Execute governed actions against business systems | Actions runtime | Delegated auth, scoped agent permissions, runtime policies, tool execution, audit evidence |
| Control both inference and business-system actions | A platform spanning both paths, or complementary layers | Model-facing controls and downstream action controls across the complete request path |
A platform can support more than one path. Evaluate its enforcement, execution, and audit capabilities rather than relying on its category name. For a deeper look at the architecture, see Arcade’s AI gateway glossary.
AI gateways compared
Model routing and LLM gateways: Quick comparison
| Product | Best for | Deployment | Key differentiation | Pricing model | Primary tradeoff |
|---|---|---|---|---|---|
| OpenRouter | Unified marketplace API and fallback across hosted models | Hosted marketplace proxy | Standardizes requests across varied provider APIs | Pay-as-you-go per token | Hosted proxy introduces a third-party data path. Verify provider routing, retention policies, and residency requirements |
| LiteLLM | Self-hosted cost tracking across OpenAI-standard endpoints | Self-hosted or managed proxy | Translates APIs from 100+ providers into OpenAI format | Free open-source; paid enterprise tier | Advanced enterprise identity features require the paid tier |
| Bifrost | High-throughput, self-hosted inference routing | Self-hosted developer infrastructure | Provider-agnostic inference routing with caching, intelligent retries, and self-hosted deployment | Free open-source; custom enterprise tier | Balancing open-source self-hosting versus enterprise capabilities |
| Vercel AI Gateway | Frontend teams in the Vercel ecosystem | Managed edge network | Native integration with Vercel AI SDK and edge deployment | Free allowance; token-based billing | Managed service with no self-hosted deployment option. OpenAI-compatible API supports integrations beyond the Vercel AI SDK |
| Cloudflare AI Gateway | Global applications needing edge routing | Managed edge network | Edge-native traffic management without deploying a Worker | Free core features; paid logs/guardrails | Relies on exact-request matching rather than semantic matching for caching |
| Portkey (Prisma AIRS AI Gateway) | Model routing, observability, security controls, and MCP governance | SaaS or self-hosted | Trace-level debugging, gateway-level controls, and guardrail enforcement | Tiered subscription plus usage | Verify downstream delegated auth, agent-scoped permissions, governed execution, and audit depth for production agent actions |
MCP gateways and enterprise agent action platforms: Quick comparison
| Product | Best for | Resources and traffic controlled | Authorization and execution model | Deployment model | Pricing model | Evaluation consideration |
|---|---|---|---|---|---|---|
| Arcade.dev | Agents taking governed actions across business systems, from single-user workflows to enterprise deployments | Agent-optimized tools, custom tools, integrated MCP servers, and governed tool execution | Delegated user auth, scoped agent permissions, per-action policy, credential isolation, reliable execution, and audit evidence | Arcade Cloud, VPC/private cloud, on-premises, and air-gapped | Free tier; Team $25/month plus usage; custom Enterprise | Model-provider routing and inference caching are separate concerns. Add a model gateway when needed |
| Kong AI Gateway | Consolidating API management and AI traffic in one control plane | API requests, model proxies, and gateway plugins | Consumer identity, claim/group-based access controls, gateway policies, and tool access rules. MCP actor-token exchange remains Tech Preview | Hybrid: Konnect-managed control plane and customer-operated data plane | Usage-based and control-plane charges; custom Enterprise | AI MCP OAuth2 plugin and token exchange are in Tech Preview |
| TrueFoundry | ML teams combining model workflows with centralized MCP access | Custom model training, evaluation, and MCP tools | Per-user OAuth flows and shared application identities | Cloud or on-premises | Free Developer; Pro $25/user/month plus usage; custom Enterprise | MLOps-centric operating model tied to specific infrastructure requirements |
| Amazon Bedrock AgentCore | Teams building agent infrastructure within AWS-managed services | Agent runtime workloads, gateway targets, tools, and supported model integrations | AgentCore Gateway, Identity, and Cedar-based Policy for configurable authentication, delegation, and tool-call enforcement | AWS managed service | Usage-based pricing across selected AgentCore services | Requires configuring multiple AWS services and validating the complete downstream action path |
| Composio | SaaS application integrations through managed connected accounts | Pre-built API integrations and connected accounts | Per-user connected accounts, managed credentials, scoped Sessions, policies, approvals, and execution logs | Cloud; enterprise self-hosting | Free tier; Pro $29/month with usage credits and metered charges | Verify agent-scoped authorization, per-action runtime policies, execution reliability, and audit evidence for production workflows |
| Obot | Teams evaluating an open-source agent and MCP platform with model and tool gateway capabilities | Conversational framework and MCP proxy | Per-user authentication, OAuth token exchange, and gateway filtering | Self-hosted or Obot Cloud | Free Community edition with user/device limits; custom Enterprise | Validate end-to-end delegated action controls and execution governance |
| Stacklok ToolHive | Teams building and operating MCP server infrastructure using open-source, self-managed tooling | MCP server lifecycle and developer automation | Open-source MCP infrastructure; Enterprise Connector Gateway supports per-user OAuth and token exchange | Self-hosted; Enterprise private and air-gapped deployment | Free Apache 2.0 OSS; custom Enterprise pricing | Verify OSS versus Enterprise capabilities and the downstream authorization and execution path |
Model routing and LLM gateways
OpenRouter: Hosted model marketplace and fallback
Best for
Teams that want a unified marketplace API and fallback across hosted open-source and proprietary models.
Overview
OpenRouter is a model marketplace and unified API that standardizes requests across providers. It’s a hosted proxy for inference traffic routing.
Capabilities that matter
OpenRouter supports configured provider fallback, passes through provider prompt caching, and offers opt-in response caching for eligible identical requests. (Opt-in response caching and provider prompt caching are functionally different.) Its provider-routing controls let you restrict eligible providers or set a preferred order. Provider-level failover stays within the selected model, while model-level fallback requires a configured backup-model list. You also get spending limits and model allowlists.
Tradeoffs and limits
OpenRouter is a hosted proxy, not a self-hosted gateway. You should evaluate its provider-routing policies, data retention settings, data residency options, and third-party processing requirements before routing sensitive workloads through it.
Pricing
Pay-as-you-go based on model token usage with a 5.5% platform fee on credit purchases. Enterprise customers can negotiate pricing terms and fee discounts. See OpenRouter pricing for details.
LiteLLM: Self-hosted model routing and cost controls
Best for
Self-hosted deployments that need cost tracking and routing across multiple LLMs using the OpenAI standard format.
Overview
LiteLLM is an open-source proxy that provides OpenAI-standard API translation, model routing, and Model Context Protocol (MCP) capabilities.
Capabilities that matter
LiteLLM translates APIs from over 100 providers into OpenAI format. It enforces budgets and rate-limiting per API key, runs fallback chains to maintain availability during provider outages, manages cache configurations, and includes basic MCP capabilities for server proxying.
In its proxy routing path, retries target another deployment in the same model group before configured fallbacks move the request to a different group. Virtual keys can restrict model access and track spending by key, user, or team.
Tradeoffs and limits
Production auth and spend tracking on self-hosted LiteLLM require PostgreSQL for keys and spend data. Multi-instance deployments also need Redis for shared caching and rate-limit state. SSO and SCIM require the paid enterprise tier.
Pricing
Free open-source edition. Paid enterprise editions bill based on deployment scale and feature set.
Bifrost: High-throughput, self-hosted inference routing
Best for
Platform teams that need customizable developer infrastructure for AI routing, MCP integrations, and enterprise access.
Overview
Bifrost is an open-source (and enterprise) developer infrastructure tool for model routing, MCP client and server functions, and enterprise identity integration.
Capabilities that matter
Bifrost gives you a provider-agnostic API, token caching, and intelligent retries for high-volume inference traffic. Its router groups deployments under public model names and supports configurable traffic strategies, circuit breakers, and bounded retries. Fallback chains kick in after the eligible deployments for a public model are exhausted. The Enterprise edition adds enterprise identity controls, audit features, and unified MCP functions.
Tradeoffs and limits
You’re taking on self-hosting operations. The choice between the open-source and Enterprise editions determines which capabilities and identity controls you get.
Pricing
Open-source is free. Enterprise pricing scales with usage and deployment requirements.
Vercel AI Gateway: Model routing for Vercel applications
Best for
Frontend teams already using the Vercel ecosystem and Vercel AI SDK.
Overview
Vercel AI Gateway is a managed gateway that routes model requests across providers and supports provider-specific prompt caching for Vercel applications.
Capabilities that matter
It runs at the edge and integrates with the Vercel AI SDK. Provider controls support explicit provider ordering and filtering. Model fallback configuration tries backup models after the available providers for the primary model fail.
Tradeoffs and limits
Vercel AI Gateway integrates natively with the Vercel AI SDK but also offers an OpenAI-compatible API. If you’re on another stack, evaluate the integration effort.
Pricing
Charges for model usage at per-token rates with no gateway markup. The free tier includes a $5 monthly allowance, which ends when you move to the paid credit tier. See Vercel AI Gateway pricing for details.
Cloudflare AI Gateway: Edge-based inference controls
Best for
Global applications that need edge routing and rate limiting integrated directly into the network proxy.
Overview
Cloudflare AI Gateway is an edge-native service on Cloudflare’s global network for AI traffic management and caching. You don’t need to deploy a Worker to use it.
Capabilities that matter
You get exact-request caching and rate limiting. You can configure provider or model fallbacks that trigger after an error or timeout, or build routing flows with Dynamic Routing. It acts as a single control plane for token usage and request metrics at the edge.
Response caching is off by default. When enabled, the default cache key uses the provider, endpoint, model, provider auth header, and full request body, so changing any of those inputs creates a separate cache entry.
Tradeoffs and limits
Logging is built in, but retention and pricing depend on when the account created its first gateway: new accounts use Workers Logs, earlier accounts use Legacy Logs. Caching relies on exact-request matching, not semantic matching.
Pricing
Core analytics, caching, and rate limiting are free. Charges can apply to Workers Logs for new gateways, Logpush exports, Workers AI Guardrails inference, and credits through Unified Billing.
Portkey (Prisma AIRS AI Gateway): Model routing and MCP governance
Best for
Teams that need model routing, granular observability, trace-level debugging, compliance guardrails, and centralized MCP access controls.
Overview
Portkey (now part of Palo Alto Networks’ Prisma AIRS AI Gateway) provides model routing, observability, security controls, and MCP gateway capabilities. Its MCP Gateway includes enterprise identity integration, tool-level access controls, and runtime policy enforcement.
Capabilities that matter
Portkey manages token usage, caches responses, routes configured failover traffic, and enforces compliance guardrails. Its MCP Gateway supports OAuth 2.1, API-token and header auth, enterprise identity-provider integration, and policy controls for MCP traffic. It also supports per-user OAuth refresh and a private data-plane option.
Tradeoffs and limits
Portkey spans model and MCP traffic. For production agents acting across enterprise systems, you need to verify how the acting user’s downstream application permissions intersect with the agent’s separately scoped permissions, where actions execute, how credentials are managed, and what execution evidence is retained. That evaluation goes beyond gateway-level authentication, tool-level RBAC, and prompt guardrails.
Pricing
The free Developer plan records up to 10,000 logs per month. Production starts at $49/month with request allowances and overages. Enterprise pricing is custom.
MCP gateways and enterprise agent action platforms
Arcade.dev: Actions runtime for governed agent execution
Best for
Teams building agents that take actions across production systems and need secure tool access, reliable execution, and centralized governance. That includes single-user agents, multi-user enterprise apps, background agents, and MCP-enabled clients.
Overview
Arcade.dev is an actions runtime for enterprise AI agents. It gives you one control point to enforce security and governance policies on agent actions, execute reliably across business systems, and govern production agents and tools from a central place.
Arcade works with any model, framework, or client. MCP is supported but not required. Free and Team plans run on Arcade Cloud. VPC/private cloud, on-premises, and air-gapped deployments are available on the Enterprise plan. Arcade maintains a SOC 2 Type II attestation.
Capabilities that matter
Enforce: On every action executed through Arcade, the runtime evaluates the intersection of the user’s existing permissions and the agent’s separately scoped permissions. It manages the OAuth credential lifecycle (including refresh and rotation) while keeping credentials isolated from the LLM. Contextual Access adds tool-visibility, pre-execution, and post-execution policy hooks that can connect your existing DLP, SIEM, and compliance infrastructure to agent actions.
Execute: Arcade ships a catalog of more than 8,000 agent-optimized tools out of the box, built for natural-language intent rather than raw API structures. You can build custom tools or host and integrate existing MCP servers. Arcade’s MCP Gateway federates tools for MCP clients under the same governed action path, while the runtime supports parallel execution, intelligent retries, and automatic failover.
For integrated remote MCP servers, Arcade governs routed tool calls while the remote servers and downstream applications stay responsible for their own operations and business rules. Actions that bypass Arcade are outside its runtime enforcement boundary.
In an eight-query Attio MCP toolkit benchmark, Arcade’s implementation returned 7,426 response tokens compared with 747,083 for the compared Composio implementation. That’s roughly 100x fewer response tokens for that specific workload and those implementations.
For large MCP Gateway tool catalogs, Arcade’s Tool Recommendation retrieves and ranks relevant tools from the gateway’s allowed set without loading every definition into context. The agent makes the final selection, and Arcade applies the same runtime enforcement and audit controls to whatever gets selected. Standard MCP clients don’t need an Arcade-specific plugin.
In a modeled single-request session against a fixed baseline of 100 tools, Tool Recommendation reduced input-token costs by up to 37% (excluding output tokens). You can disable it for small, fixed tool sets.
Govern: Arcade centrally manages agents, tools, access, versions, and runtime configuration. Audit Logs record administrative and configuration changes. Tool Executions record runtime calls and outcomes: the tool and version, the represented user, inputs, outputs, status, attempts, errors, and timing. Sensitive payload access is restricted by role.
Together, these records provide fine-grained, immutable auditability. You can determine which agent took what action, on behalf of which user, against which system, when, and with what outcome.
Tradeoffs and limits
Arcade focuses on governed tool access and reliable action execution. It doesn’t do model-provider routing or inference caching. If you need those, run Arcade alongside a model gateway. Delegated user access requires configuring the relevant identity, authorization, and consent flows.
Pricing
The free tier covers 2,000 auth events and 2,000 tool calls per month. Team is $25/month plus $0.10 per auth event and $0.01 per tool call. See Arcade’s pricing page for details. Enterprise pricing is custom.
Kong AI Gateway: API management and AI traffic control
Best for
Enterprise platform engineering teams already on Kong that want to consolidate traditional API management and AI traffic in one control plane.
Overview
Kong AI Gateway provides centralized management for LLM, MCP, and agent-to-agent traffic through a Konnect-managed control plane and customer-operated data plane. It supports model routing, API access controls, gateway policies, and proxying of upstream MCP servers. Its AI Gateway entity model is separate from Kong Gateway Enterprise’s self-hosted deployment configurations.
Capabilities that matter
Kong provides centralized API management, model routing, token-aware rate limiting, consumer identities, claim- and group-based access controls, and tool-level policy enforcement. The exact capabilities depend on the product, edition, and version you select. Its AI MCP OAuth2 plugin, including actor-token exchange, remains in Tech Preview and isn’t recommended for production.
Tradeoffs and limits
The AI MCP OAuth2 plugin is in Tech Preview and isn’t intended for production use. (That applies to the specific plugin, not Kong Gateway as a whole.) If you need production delegated authorization, verify the release status, licensing, token-exchange behavior, and enforcement capabilities of the exact Kong components you plan to deploy.
The current AI Gateway entity model requires a Konnect-managed control plane. If you need a fully customer-operated control plane, you’ll need to evaluate the separate Kong Gateway Enterprise offering and its corresponding capabilities and licensing.
Pricing
Kong AI Gateway pricing is distinct from Kong’s free open-source gateway and its separate Gateway Enterprise offering. Current pricing includes control-plane, proxied-model, and API-request charges. Enterprise pricing is custom.
TrueFoundry: Model workflows and centralized MCP access
Best for
ML platform teams combining custom model training, evaluation, and deployment workflows with centralized MCP access.
Overview
TrueFoundry is an MLOps platform that handles model training, LLM Gateway routing, and centralized MCP access across cloud and on-premises environments.
Capabilities that matter
TrueFoundry supports per-user authorization-code OAuth flows and shared client-credentials OAuth flows for MCP access. The client-credentials flow uses a shared application identity rather than an individual user’s delegated identity. TrueFoundry also provides integrated tracing.
Tradeoffs and limits
You’ll need to evaluate TrueFoundry’s MLOps-centric operating model, its multi-surface gateway capabilities, and the identity configuration requirements for your target VPC or cloud footprint.
Pricing
The free Developer plan supports up to three users and 10,000 requests per user per month. Pro costs $25/user/month and includes 20,000 pooled gateway requests or tool calls per user, with additional usage billed at $20 per 100,000 requests. Enterprise, Agent Gateway, and Agent Harness use custom pricing. Model-provider and managed third-party guardrail charges are separate. See TrueFoundry’s pricing page for details.
Amazon Bedrock AgentCore: Agent execution within AWS
Best for
Organizations building agent infrastructure within AWS-managed services.
Overview
Amazon Bedrock AgentCore provides AWS-managed services for hosting agents, connecting tools, managing identities, and enforcing tool-call policies. AgentCore Runtime supports multiple models and frameworks, while Gateway, Identity, and Policy are separate services that control access to tools and downstream systems.
Capabilities that matter
AgentCore Gateway authenticates callers. AgentCore Identity handles outbound credentials and on-behalf-of token exchange. Policy in AgentCore evaluates Cedar policies for Gateway tool calls. The service supports OAuth and user-delegated authorization flows within the AWS service model and generates execution logs.
Tradeoffs and limits
Governed action workflows require configuring and connecting AgentCore Gateway, Identity, Policy, IAM roles, and the relevant target services. That’s a lot of moving parts.
Gateway-level policies don’t automatically govern direct AgentCore Runtime invocations. AWS provides configurations that restrict Runtime access to Gateway, and you need to apply those restrictions when the gateway is supposed to be the single enforcement path. Validate identity flows, authorization decisions, credential handling, tool execution, and audit evidence across the configured services.
Pricing
AgentCore meters Gateway requests, Policy authorization, and other selected services separately. Identity has no additional charge when used through AgentCore Runtime or Gateway. Standalone Identity requests and model use can carry separate charges.
Composio: Pre-built SaaS integrations for agents
Best for
Developers connecting AI agents to SaaS applications through managed account connections, OAuth flows, and prebuilt tool integrations.
Overview
Composio is an MCP gateway and integration wrapper that provides prebuilt SaaS integrations, managed connected accounts, and tool execution capabilities for AI agents. Its Sessions layer supports tool discovery, access scoping, authentication, and execution workflows across connected applications.
Capabilities that matter
Composio provides prebuilt integrations, per-user connected accounts, managed OAuth credentials, scoped tools, session controls, execution APIs, policies, approvals, and execution logs. Enterprise capabilities include identity integration, SSO, SCIM, role-based controls, and self-hosting. Its managed connections and session-based execution model support application integrations across multiple users and agent workflows.
Tradeoffs and limits
Composio Sessions make the toolkit catalog discoverable by default unless you explicitly restrict access. Production deployments should configure toolkit and tool allowlists to limit which integrations agents can invoke.
Custom Tools and Toolkits and Custom MCP are experimental. Custom MCP connects to externally hosted, publicly reachable HTTPS servers rather than hosting them, and changes to tool definitions require manual resync.
You should also evaluate per-action authorization semantics, runtime policy context, execution reliability, and audit coverage for your intended workflows.
Pricing
The free tier includes 100,000 tool calls and 50,000 triggers per month. Pro starts at $29/month with a usage credit. Overages and selected add-ons are metered. Connections using your own OAuth app are free; Composio-managed apps have different allowances.
Obot: Agent deployment with MCP proxying
Best for
Teams evaluating an open-source agent and MCP platform with model and tool gateway capabilities, available self-hosted or hosted.
Overview
Obot is an open-source agent and MCP platform with MCP proxying and LLM Gateway routing for conversational and autonomous AI systems.
Capabilities that matter
Obot supports per-user authentication, authorization, OAuth token exchange, gateway filtering, and audit logging alongside its LLM Gateway routing capabilities. Its MCP Gateway architecture separates protocol forwarding from the server shim that applies authorization, token exchange, and audit logging.
Tradeoffs and limits
Evaluate Obot’s delegated authorization, runtime policy controls, tool execution, and audit configuration against your production requirements. The deployment model determines the operational work involved in maintaining, upgrading, and securing the platform.
Self-hosting alone doesn’t tell you how downstream actions are authorized and governed.
Pricing
Obot Community is free with limits of 100 users and 100 devices. Obot Cloud is also available. Enterprise pricing is by quote.
Stacklok ToolHive: MCP server governance for automation
Best for
Teams building and operating MCP server infrastructure using open-source, self-managed tooling.
Overview
ToolHive is an open-source, Kubernetes-native platform for deploying and managing MCP servers and related developer tooling. Stacklok Enterprise adds centralized management, identity integration, access controls, audit, and customer-operated deployment options.
Capabilities that matter
Stacklok Enterprise supports centralized MCP server management, identity integration, access controls, audit logging, and Connector Gateway functionality. Connector Gateway supports per-user OAuth authorization and token exchange for downstream MCP connections. Audit events capture operation metadata, with configurable payload logging.
Tradeoffs and limits
ToolHive’s open-source capabilities and Stacklok Enterprise Connector Gateway have different feature and licensing boundaries. Verify which deployment supports your required delegated auth, agent-scoped policy enforcement, execution controls, and audit evidence.
Self-hosted deployment alone doesn’t mean these controls are configured across the downstream action path. Stacklok Enterprise supports per-user downstream authorization through Connector Gateway. Other ToolHive deployment paths require separate configuration and validation. Enterprise AI Gateway and Connector Gateway components must be enabled and configured for your intended deployment.
Pricing
ToolHive is free under Apache 2.0. Stacklok Enterprise uses custom pricing. Customer infrastructure and support are separate cost components.
How model routing and agent actions work together
Model gateways and actions runtimes can operate together in the same agent workflow. The model gateway controls model-provider access, inference routing, caching, and failover. The actions runtime enforces permissions and policies when agents interact with external business systems.
Some products offer both paths. Don’t assume that MCP or model-routing support means the complete control and execution requirements are covered. Verify it.
Here’s a concrete example. A sales agent is asked to update a Salesforce opportunity and notify the account team in Slack.
The model gateway routes the inference request to an approved LLM. When the model determines which business actions to take, it generates the appropriate tool calls.
Those tool calls execute through Arcade’s actions runtime. Arcade checks the user’s permissions and the agent’s separately scoped permissions, applies configured runtime policies, manages credentials outside the LLM, and executes the permitted actions using Arcade tools or integrated MCP servers.
If the agent tries to modify a record outside its authorized scope, Arcade denies the action before execution. Tool execution records show what was permitted, what failed, and why.
The model gateway controls inference traffic. Arcade governs the actions taken against business systems. You can combine both layers without tying the agent to a particular model or orchestration framework.
How to choose an AI gateway
Evaluating the right infrastructure means mapping the request path and testing the actual controls:
- What traffic needs to be controlled? Figure out whether the workload requires model inference routing, MCP tool access, agent-to-agent communication, or governed actions against business systems.
- Will agents read data or take actions? Agents that modify records, send messages, initiate transactions, or operate across multiple user accounts need authorization and policy checks on each action.
- How are user and agent permissions enforced? Verify whether the gateway preserves the acting user’s downstream permissions, independently scopes what the agent can do, and applies authorization and policy checks at execution time.
- Where do actions execute, and how reliable is execution? Compare raw API proxying with agent-optimized tools, credential lifecycle management, retries, failure handling, and controlled execution.
- How are tools discovered and governed? Look at tool catalogs, custom tools, MCP server integration, tool recommendation, versioning, and centralized access controls.
- What audit evidence is available? Determine whether the platform records administrative changes separately from runtime tool calls, including user identity, tool version, execution attempts, errors, and outcomes.
- Where must the infrastructure run? Compare cloud, VPC/private cloud, on-premises, and air-gapped deployment options alongside operational cost and maintenance responsibilities.
Tool quality matters, not just tool connectivity. A gateway can expose thousands of tools without making those tools reliable for agents. In a ToolBench rescan of more than 90,000 open-source MCP servers, only 260 servers (0.28%) received an A grade under the revised scoring rubric. That describes the evaluated open-source server population, not all commercial MCP platforms. Tool descriptions, input schemas, recovery guidance, and argument constraints matter as much as compatibility and gateway performance.
For hosting-specific requirements, compare the separate responsibilities covered by MCP server deployment platforms.
How to test an AI gateway before choosing one
Test shortlisted platforms against equivalent workloads and deployment requirements. Model routing and governed agent actions need different validation scenarios.
| Test | What to verify |
|---|---|
| Model-provider failure | Whether the gateway retries, fails over, or returns an error for timeouts, rate limits, and provider failures |
| Inference cost controls | Whether token budgets, caching, and per-model usage reports reflect actual requests and charges |
| Delegated user authorization | Whether two users with different downstream permissions get appropriately different tool access |
| Agent scope enforcement | Whether an out-of-scope action is denied even when the user has broader permissions |
| Tool execution failure | Whether retries and recovery work correctly without accidentally repeating non-idempotent actions |
| Audit and observability | Whether the records identify the relevant user, agent, resource, policy decision, execution outcome, and errors |
Identify which controls run at the gateway, which run in downstream applications, and which require additional services. For an agent action test: authenticate two users, permit one legitimate action, deny an out-of-scope action, trigger an execution failure, and inspect the resulting evidence.
Conclusion
The right AI gateway depends on which part of your AI runtime needs control.
Model gateways manage inference traffic: provider routing, token costs, failover. Agent gateways and actions runtimes address how agents connect to tools and perform operations against external systems.
For model routing, shortlist products based on deployment requirements, provider coverage, observability, and cost controls. OpenRouter, LiteLLM, Bifrost, Vercel AI Gateway, Cloudflare AI Gateway, and Portkey address different combinations of these.
For agents taking actions across enterprise systems, Arcade.dev provides a unified actions runtime: delegated user auth, scoped agent permissions, per-action policy enforcement, agent-optimized tools, reliable execution, and centralized governance.
Test a real workflow with multiple users, a permitted action, a denied action, and an execution failure. Verify how each platform enforces policies, manages credentials, handles execution, and records outcomes.
If you also need multi-model inference routing, Arcade works alongside a model gateway without changing the agent’s model or orchestration framework.
Try Arcade’s actions runtime for free.
FAQ
When does an AI agent need an actions runtime?
When it needs to execute governed operations against external systems: updating CRM records, sending messages, creating support tickets, triggering workflows. An actions runtime manages delegated auth, enforces scoped agent permissions and runtime policies, and provides reliable execution and auditability. It can run independently or alongside a model gateway. For related connectivity choices, see the AI agent integration platforms comparison.
How do AI gateways handle per-user authentication for agents?
It depends on the gateway. Model gateways commonly authenticate inference requests with API keys. For actions in business systems, verify that the product can use the individual user’s delegated identity and enforce authorization on each tool execution.
What is the difference between an AI gateway and an MCP gateway?
An AI gateway is a broad buying label covering infrastructure that manages access to AI models and external tools. An MCP gateway specifically brokers AI client connections to tools using the Model Context Protocol. Supporting MCP or generic OAuth doesn’t automatically establish secure downstream per-user authorization for enterprise applications. For an MCP-specific shortlist, see the enterprise MCP gateways comparison.
Which AI gateway should I choose for routing across multiple LLM providers?
It depends on your deployment and operational needs. LiteLLM suits self-hosted routing and cost controls. Portkey focuses on observability and prompt guardrails. OpenRouter offers a hosted model marketplace. Verify each product’s failover and data-handling options against your requirements.
Which AI gateways are open source or self-hosted?
LiteLLM and Bifrost offer open-source model-routing software. Obot and Stacklok ToolHive provide open-source agent and MCP infrastructure. Other products support customer-operated deployments through specific commercial plans. Compare licensing, feature availability, and operational responsibilities separately from source-code availability.
Can one AI gateway handle both LLM routing and MCP tool access?
Yes. Some platforms manage model inference and MCP traffic in one product. But supporting both traffic types doesn’t automatically give you delegated user authorization, separately scoped agent permissions, reliable downstream execution, or complete auditability. Test each requirement on the actual request path.
Do AI gateways reduce LLM costs?
They can, through caching, routing, token budgets, and provider selection. Actual savings depend on workload characteristics, cache hit rates, provider prices, and gateway fees. Agent-optimized tools and dynamic tool recommendation can also reduce context usage, but evaluate those savings separately from model-routing costs.
How much does an AI gateway cost?
AI gateways can start at $0 with free tiers or open-source software. Paid model gateways may charge per token, request, or platform fee. MCP gateways and actions runtimes may bill for tool calls, auth events, users, or hosting. Total production costs also include model usage, infrastructure, logging, and any required security add-ons.
