An AI gateway is an infrastructure layer that sits between AI applications or agents and the AI models they use. It gives applications a shared interface for accessing model providers while centralizing authentication, credentials, routing, usage limits, reliability controls, telemetry, and cost tracking.
AI gateways initially focused on model access. Instead of every application integrating separately with OpenAI, Anthropic, Google, self-hosted models, or other providers, applications could call one gateway and let the gateway handle the provider-specific connection.
The category is expanding as AI applications become agentic. Agents now call enterprise tools through the Model Context Protocol (MCP), delegate work to other agents, and operate across more runtime resources than models alone. A broader AI gateway can therefore govern model access, MCP server and tool access, and, in some implementations, agent-to-agent traffic.
Some products still use “AI gateway” and “model gateway” interchangeably. This article uses the broader definition while distinguishing the model-facing, tool-facing, and agent-facing parts of the architecture.
A single gateway does not have to handle every type of traffic. An organization can run separate model and MCP gateways while sharing identity, policy, configuration, and telemetry across them.
Key takeaways
- An AI gateway creates a shared control point for AI traffic. It sits between applications or agents and resources such as models and MCP servers.
- Its core functions include access control, routing, credential management, usage limits, monitoring, and cost tracking.
- AI gateways can be centralized logically without running as one centralized process. Organizations can use multiple distributed or specialized gateway instances.
- A gateway only controls the traffic and context it can see. It complements downstream authorization, agent safeguards, observability, and broader AI governance rather than replacing them.
Why do organizations use AI gateways?
AI gateways become useful when teams need to apply the same controls across multiple AI applications, models, agents, or tools.
Without a shared gateway, application teams may independently solve the same infrastructure problems:
- managing provider and service credentials
- supporting different provider APIs
- authenticating applications, agents, and users
- controlling which models and tools they can access
- applying request, token, concurrency, or budget limits
- routing requests and handling failures
- logging and tracing usage
- attributing model and tool costs
The problem is not only duplicated engineering. Independent integrations can also create different security, routing, and operational policies across teams.
Centralize access to multiple model providers
A company may use models from several providers alongside internally hosted models.
Without a gateway, every application may need to understand each provider’s API, credentials, model names, error behavior, and usage limits.
A gateway provides a common access layer. Applications can request an approved model or logical model alias while the gateway resolves that request to an eligible provider or deployment.
In 2024, Uber described more than 60 LLM use cases across teams before consolidating model access through its GenAI Gateway. At the time, nearly 30 teams were sending roughly 16 million queries per month through the system.
Control agent access to tools
Agents create a similar problem for enterprise tools.
MCP gives agents a standard protocol for discovering and invoking tools, but organizations still need to decide which MCP servers an agent may reach, which tools it may discover, whether a particular invocation is authorized, which downstream credential should be used, and how the action should be logged.
An MCP gateway can become that shared control point.
DoorDash uses this pattern for its internal Agent Gateway. In July 2026, it reported more than 200 registered MCP servers, thousands of tools, more than 30 agents and services, and millions of tool calls per week.
Apply runtime policy consistently
A shared gateway gives platform and security teams a common place to enforce rules such as:
- which models may handle a workload
- which providers or regions are eligible
- which MCP servers an agent may access
- which tools an agent may discover
- whether a specific
tools/callis authorized - how downstream credentials are obtained
- how much a consumer may use
- what telemetry should be recorded
The gateway does not replace the underlying systems that make these decisions possible. Identity, application context, and downstream authorization still matter.
When do you need an AI gateway?
Not every AI application needs a gateway.
Direct access can remain simpler when one application uses one or a small number of resources and there is no need for shared policy, centralized credentials, provider abstraction, or organization-wide usage tracking.
An AI gateway becomes more useful when one or more of these conditions appear:
| Trigger | Why a gateway helps |
|---|---|
| Multiple model providers | Gives applications one access layer instead of separate provider integrations |
| Several applications or agents | Applies common access, routing, and usage policy across consumers |
| Centralized credential requirements | Keeps provider or downstream credentials out of individual applications and model context |
| Shared rate or token limits | Enforces usage rules across applications, users, teams, or agents |
| Reliability requirements | Centralizes retries, fallback, health-aware routing, or load balancing |
| MCP adoption | Provides a controlled path for MCP server access, tool discovery, tool authorization, and credential brokering |
| Centralized observability | Records model, tool, policy, latency, and usage telemetry consistently |
| Cost allocation | Attributes model and tool consumption to applications, teams, users, or agents |
The decision is less about reaching a specific scale and more about whether shared control is becoming easier to operate than independent connections.
How does an AI gateway work?
An AI gateway sits in the runtime path between a caller and an AI resource.
What happens at the gateway depends on the traffic. A model request, an MCP tool call, and an agent-to-agent task expose different information and require different controls.
Model request flow
A model request commonly follows this flow:
- An application sends a model request to the gateway.
- The gateway authenticates the caller and checks which models it may use.
- It applies request, token, concurrency, or other usage policies.
- It resolves the requested model or logical alias to one or more approved backends.
- It selects an eligible backend according to requirements such as region, availability, latency, capacity, or cost.
- It supplies the provider credential, translates the request where required, sends it to the provider, and records usage, latency, errors, and policy decisions.
A common API does not make models interchangeable. Providers expose different capabilities, tool interfaces, context limits, output formats, and model behavior. Provider abstraction reduces integration work, but applications may still need access to provider-specific features.
Fallback models also need to be evaluated for the workload using them.
MCP and tool request flow
Consider an internal support agent that needs customer data through an enterprise MCP server.
Instead of connecting directly to every MCP server, the agent connects through an MCP gateway:
- The agent authenticates to the gateway.
- The gateway determines which MCP servers and tools the agent or acting user may access.
- A
tools/listrequest returns only the approved tools visible to that caller. - When the agent sends a
tools/call, the gateway performs a separate authorization check for that invocation. - The gateway obtains or exchanges the appropriate downstream credential and sends the request to the MCP server.
- It records the caller, MCP server, tool, authorization decision, latency, and outcome.
Tool discovery and tool authorization solve different problems. An agent being able to see a tool does not automatically mean it is allowed to invoke it.
This distinction becomes important when an MCP server exposes tools with different risk levels. A support agent might be allowed to discover and invoke get_customer, for example, while delete_customer requires a different policy or is hidden entirely.
Credentials also cross separate authorization boundaries. A token accepted by the MCP gateway is not automatically valid for Salesforce, GitHub, or another downstream service. The gateway may need to obtain, exchange, or broker a credential intended for that resource.
Agent-to-agent request flow
A gateway can also sit between agents.
For example, a support agent might delegate a billing investigation to another agent. An A2A-aware gateway can authenticate the calling agent, determine whether it may access the target agent, route the task, and record the interaction.
The target agent still owns task execution and state.
The gateway also cannot create identity or delegation context that was never supplied. If the billing agent needs to know which user authorized the original request, that information has to be established and propagated by the participating applications and identity systems.
Three AI gateway traffic paths
These paths explain why the term “AI gateway” is broader than model routing alone. The common architectural pattern is a runtime intermediary that understands enough about the resource and operation to apply AI-specific controls.
Control plane and data plane
AI gateways often separate configuration from request processing.
The data plane handles live traffic. It receives requests, applies active policy, routes them, and records telemetry.
The control plane manages resource registrations, routing rules, policies, consumer identities, and secret references.
These are logical roles rather than requirements for separate products or processes.
An organization can run gateway instances across multiple regions while managing their configuration centrally. Some architectures allow a data plane to continue using its last known configuration when the control plane is temporarily unavailable.
That improves availability, but creates a tradeoff. A new policy or revoked permission may not reach the data plane until communication is restored.
How is an AI gateway different from an API gateway?
An AI gateway builds on many of the same architectural patterns as an API gateway, but it understands AI-specific resources, usage units, protocols, and routing decisions.
| Area | API gateway | AI gateway |
|---|---|---|
| Primary traffic | HTTP, REST, gRPC, and general APIs | Model APIs, MCP traffic, and potentially agent traffic |
| Routing target | API service or backend | Model/provider, MCP server/tool, or agent |
| Rate limiting | Commonly requests per unit of time | Can include requests, tokens, concurrency, model usage, or budgets |
| Credentials | API or service credentials | Model-provider credentials plus tool or delegated-user credentials |
| Provider abstraction | Normalizes access to APIs where configured | Can normalize common operations across multiple model providers |
| Streaming | General request/response streaming | Often needs to account for model token streaming and partial responses |
| Observability | API latency, status, requests, errors | Adds models, tokens, tools, policy decisions, estimated cost, and agent context |
| AI-specific semantics | Not inherent | Can understand model names, token usage, MCP methods, tools, or agent operations |
An existing API gateway can be extended with AI-specific capabilities. A separate product is not required simply because traffic involves AI.
The useful distinction is whether the gateway understands and controls the AI resources and operations passing through it.
What is the difference between an AI gateway, model gateway, and MCP gateway?
The terms overlap, but they describe different parts of the runtime path.
| Gateway | Primary traffic | Main responsibility |
|---|---|---|
| Model gateway / LLM gateway | Application or agent → model provider | Provider abstraction, model routing, credentials, token limits, fallback, usage telemetry |
| MCP gateway | MCP client → MCP server/tool | MCP server access, tool discovery, tool authorization, credential brokering, MCP telemetry |
| Agent gateway | Client or agent → agent | Agent access policy, identity propagation, task routing, agent interaction telemetry |
| AI gateway | Can span multiple AI traffic types | Broader runtime layer that can combine these functions with common identity, policy, administration, observability, and cost controls |
A model gateway is therefore the model-facing part of the category. An MCP gateway is the tool-facing part.
Some products use “AI gateway” to mean only the model gateway. Others use it for infrastructure that governs models, MCP servers, tools, and agents through a shared control layer.
For a product-level comparison, see 13 enterprise MCP gateway options for 2026.
What are common AI gateway use cases?
Shared model access
Several applications access approved models through a common interface rather than maintaining separate integrations and credentials for every provider.
This reduces provider-specific integration work and makes it easier to restrict models, change backends, apply quotas, and collect usage consistently.
Provider abstraction
Different model providers expose different endpoints, model identifiers, authentication methods, and request formats.
A gateway can normalize the operations that are common across providers so applications do not need separate integration code for every backend.
Provider abstraction has limits. A unified API cannot make every provider feature identical, so applications may still need provider-specific extensions for capabilities that do not map cleanly to a common interface.
Controlled MCP server and tool access
Agents access enterprise tools through an MCP gateway instead of connecting directly to every server.
The gateway can control:
- which MCP servers a caller can reach
- which tools appear during discovery
- whether a specific tool invocation is authorized
- which user or workload identity applies
- which downstream credential is used
- what MCP activity is recorded
For example, a support agent might be allowed to read a customer account but not invoke an account-deletion tool exposed by the same MCP server.
Credential management
The gateway can hold, retrieve, or exchange downstream credentials rather than exposing them in application code or model context.
The caller still needs to authenticate to the gateway. The difference is that every application does not need to independently manage credentials for every provider or enterprise service.
Usage and cost controls
Because requests pass through a common point, the gateway can associate consumption with an application, user, team, workload, or agent.
Depending on the traffic, controls may include:
- requests per time period
- model tokens
- concurrent requests
- tool invocations
- provider quotas
- estimated spend or budgets
This data can support budgeting, showback, chargeback, or enforcement.
Routing and reliability
A gateway can map a logical model name to several approved deployments and choose an eligible backend according to policy.
It can also centralize common reliability patterns such as load balancing, retries, fallback, circuit breaking, and health-aware routing.
For example, requests may need to remain within a particular region and fail over only to models approved for the same workload and data classification.
Retries still need workload-specific handling. Retrying a model call usually adds cost. Retrying a tool that changes real-world state can repeat the action.
What resources does an AI gateway manage?
An AI gateway understands resources that a generic reverse proxy does not necessarily treat as first-class objects.
Models and inference endpoints
These include chat, generation, embedding, multimodal, and other model APIs from external providers or internally hosted models.
The gateway may use information such as the model, provider, endpoint, region, capabilities, credential reference, and usage limits when routing a request.
MCP servers and tools
An MCP server exposes capabilities that AI applications and agents can discover and invoke.
A gateway may track the server endpoint, authentication method, owning team, available tools, access policies, and health information.
At the tool level, it can use information such as the tool name, input schema, approved definition, and authorization rules.
Agents
Agent-aware gateways may treat other agents as addressable resources.
Relevant information can include agent identity, endpoint, protocol, advertised capabilities, owner, and authentication requirements.
Consumers
The gateway also needs to understand who is making the request.
A consumer might be an application, workload, agent, user, team, or a combination of them. That identity can determine which resources are available and which quotas or policies apply.
AI asset registry and runtime metadata
These resources need an inventory somewhere.
Some gateways include their own registry. Others consume information maintained by a separate system.
A simplified resource model might look like this:
| Resource | Example metadata |
|---|---|
| Model | Alias, provider, endpoint, capabilities, credential reference, region, quota, cost information |
| MCP server | ID, endpoint, transport, authentication method, owner, available capabilities, health |
| MCP tool | Server, tool name, input schema, approved definition, access rules |
| Agent | Agent ID, endpoint, protocol, capabilities, owner, authentication requirements |
| Consumer | Application, workload, agent or user identity, team, permitted resources, quotas |
A registry only becomes useful for runtime control when changes can reach the gateways enforcing policy.
Listing an MCP server or model in a catalog does not by itself mean access to it is governed.
What are the core capabilities of an AI gateway?
Different implementations package these functions differently, but the broader AI gateway category commonly includes the following capabilities.
| Capability | What it controls | Example |
|---|---|---|
| Administration and policy management | How resources, consumers, and gateway rules are configured | Register a model and restrict it to approved applications |
| AI asset registry | Which AI resources are known to the gateway | Track approved model endpoints and MCP servers |
| Provider abstraction | How applications access different model providers through a common interface | Route a common chat request to an approved OpenAI, Anthropic, Google, or self-hosted backend |
| Model gateway | Access from applications or agents to model providers | Route a logical model alias to an approved deployment |
| MCP gateway | Access from MCP clients to approved MCP servers and tools | Filter tools/list, authorize tools/call, and broker downstream credentials |
| Identity and access control | Which callers can access which resources and actions | Allow a support agent to read account data but not delete it |
| Credential management | How credentials for downstream resources are supplied | Exchange delegated user authorization for a downstream OAuth token |
| Usage controls | How much a caller may consume | Apply request, token, concurrency, tool-call, or budget limits |
| Routing and reliability | Which eligible backend handles a request and what happens on failure | Route by region and fail over to another approved deployment |
| Monitoring and analytics | What happened across gatewayed traffic | Record latency, errors, tool calls, model usage, and policy decisions |
| Cost management | How consumption is attributed and estimated | Attribute token usage to an application or team |
Routing, retries, and fallback
Routing can be static or dynamic.
A gateway might route requests according to model alias, provider availability, region, capacity, latency, cost, or another configured rule.
It can also implement retries and fallback when an eligible backend fails.
Fallback should stay within the original policy boundary. An authorization failure should not cause a request to be rerouted to a backend that bypasses the policy, and a fallback model should not be assumed to behave identically to the primary model.
Tool calls require additional care because retries may repeat a real-world action.
Caching
Caching is an optional AI gateway capability.
Exact caching reuses a previous response when the relevant request matches the cache key.
Semantic caching attempts to reuse a response for requests considered similar in meaning.
Caching can reduce latency and model cost, but it introduces authorization, freshness, and isolation risks. Two similar prompts may belong to different users, reference different underlying data, or require different permissions.
Any cache therefore needs appropriate tenant boundaries, access controls, expiry, and invalidation rules.
Some gateways also provide content inspection, automated model selection, or agent-to-agent routing. These capabilities are not implemented the same way everywhere.
What can and can’t an AI gateway control?
An AI gateway is a strong enforcement point for decisions that can be made from information visible in the request path.
Its limits matter just as much.
What can a gateway authorize?
A gateway can decide whether an application, agent, or user may reach a registered resource.
When it has enough trusted context, it can also make decisions about a specific operation.
For example, an agent might be allowed to use a Salesforce lookup tool but not a tool that deletes a customer record.
Authentication alone is not enough. Knowing which agent made a request does not automatically establish which user it represents or which action that user authorized.
What can a gateway do with credentials?
A gateway can keep provider keys and OAuth tokens out of model context and application code.
It can retrieve or exchange credentials only when a request needs them, reducing the number of applications that need direct access to downstream secrets.
The application or agent still needs an approved identity when connecting to the gateway.
What traffic can a gateway inspect?
Some gateways inspect prompts, tool arguments, and responses for sensitive data or suspicious content.
This can support DLP, prompt-injection detection, and other runtime defenses.
The gateway can only inspect traffic and formats visible to it. Detection can also miss unsafe content or incorrectly block legitimate content whose meaning depends on context.
What still belongs to downstream systems?
Gateway authorization does not replace authorization in the system of record.
The gateway can decide whether an agent is allowed to attempt an action. Salesforce, GitHub, a database, or another downstream system should still enforce its own permissions when the request arrives.
An action can also pass every formal authorization check and still be inappropriate.
For example, an agent might technically be allowed to issue a refund while misunderstanding what the user asked it to do. The gateway can enforce the policies it has been given, but it cannot independently determine human intent for every action.
What happens when traffic bypasses the gateway?
A gateway only governs requests that pass through it.
If applications retain unrestricted direct access to model providers or MCP servers, gateway policy can be bypassed.
Network controls, IAM, credential access, and downstream authorization therefore remain part of the enforcement architecture.
The gateway is a control point for the traffic it mediates, not a control plane for everything an AI system does.
How does an AI gateway support observability and cost management?
A gateway sees requests from many applications and agents, making it a useful place to collect consistent operational data.
Observability and telemetry
For model traffic, a gateway can record:
- caller identity
- requested and actual model
- provider
- latency
- errors
- retries
- input and output tokens
- policy decisions
- estimated cost
For MCP traffic, it can record the caller, MCP server, method, tool, authorization result, latency, and execution outcome.
For A2A traffic, it can record source and target agents, task operations, task IDs, states, and policy decisions.
OpenTelemetry’s GenAI semantic conventions are developing shared terminology for model, agent, and tool telemetry. Some conventions for newer agent and MCP workloads are still evolving.
Gateway telemetry is only one part of AI observability.
A request can successfully pass through the gateway while the overall task still fails. An agent might call several tools successfully and still produce the wrong answer.
“The gateway returned 200” is not the same as “the user’s task succeeded.”
Gateway traces are most useful when they can be correlated with application, agent, tool, and downstream-service telemetry.
Usage and cost metering
A gateway can connect usage to the application, agent, team, or user that generated it.
For model calls, that often includes provider, model, request count, and token usage. For tools, it may include invocation volume or other service-specific usage units.
This makes the gateway useful for budgeting, quotas, showback, and chargeback.
It does not make the gateway a complete cost-accounting system.
An agent task can also consume compute, retrieval infrastructure, storage, SaaS APIs, browser sessions, and other cloud resources that never appear as model tokens at the gateway.
There is also a difference between measuring usage, estimating cost, and enforcing a budget. Token consumption may not be fully known until a request completes, and pricing varies across resources.
Cost per successful task therefore requires gateway data to be combined with provider billing, other infrastructure costs, and application-level outcome data.
What are the tradeoffs of an AI gateway?
Putting shared controls in the request path simplifies some parts of AI infrastructure but introduces new dependencies.
Latency. Authentication, policy evaluation, routing, credential retrieval, and content inspection all add work to a request. Some controls can operate while a response streams. Others may require buffering before the gateway can make a decision.
Retries and side effects. Retrying a model request usually adds latency and cost. Retrying a tool action can be more serious because it may repeat a payment, deletion, ticket creation, or other real-world action. Retry policy needs to account for the semantics and idempotency of the operation.
Caching. Caching can reduce model cost and latency, but similar requests are not always interchangeable. User permissions, tenant boundaries, underlying data, and freshness requirements can change whether a cached result is appropriate.
Shared dependencies. If many applications depend on the gateway, failures in the gateway or its identity, policy, and credential services can affect all of them. Distributed data planes and cached configuration can improve availability, but policy changes or revocations may take longer to propagate.
Salesforce’s AI inference architecture illustrates why these dependencies need optimization. Its engineering team reported reducing metadata lookup time from roughly 400 milliseconds to a few milliseconds through multi-layer caching, contributing to a reduction in P90 end-to-end latency from roughly 15 seconds to 11 seconds.
How does an AI gateway compare to related concepts?
Beyond model and MCP gateways, several adjacent categories overlap with AI gateway responsibilities.
| Concept | Main role | Relationship to an AI gateway |
|---|---|---|
| Agent gateway | Controls interactions involving agents, tools, or other agents | Overlaps with the broader AI gateway category but may not handle model traffic |
| AI governance platform | Manages AI inventory, policies, approvals, risk, evidence, and governance across the lifecycle | Can use gateways as runtime enforcement points but covers responsibilities beyond request mediation |
| AI agent management platform | Manages an organization’s population of agents across deployment environments and lifecycle stages | Can configure or integrate with runtime gateways but also covers ownership, evaluation, deployment state, cost, and lifecycle management |
The terminology is still evolving, and products do not always use these labels consistently.
The useful architectural question is which traffic and resources each component controls.
Who manages an AI gateway?
AI gateways usually sit across several existing responsibilities rather than belonging to a completely new team.
| Team | Typical responsibility |
|---|---|
| Platform or infrastructure engineering | Operates gateway infrastructure, provider integrations, routing, availability, and deployment |
| AI and application engineering | Integrates applications and agents and defines workload requirements |
| Security and IAM | Defines authentication, authorization, credential, and access requirements |
| FinOps | Uses gateway usage data for allocation, budgets, and cost analysis |
| Governance and risk | Defines organizational requirements that runtime gateway policies may help enforce |
The exact ownership model depends on the organization.
A platform team might operate the gateway while security owns access policy and application teams remain responsible for deciding which models or tools their workloads require.
How are AI gateways evolving?
AI gateways began largely as a way to centralize model-provider access.
Applications could call one interface while the gateway handled credentials, provider abstraction, model routing, token limits, retries, telemetry, and cost information.
Agents are expanding the scope.
Once agents began calling enterprise tools through MCP, organizations needed similar controls for MCP server access, tool discovery, invocation authorization, credential brokering, and MCP observability.
Agent-to-agent protocols add another traffic surface. An agent may now call a model, invoke an MCP tool, and delegate part of a task to another agent within the same workflow.
Protocols such as MCP and A2A make these interactions easier for gateway infrastructure to understand because they expose more information about the target resource and operation.
That does not solve the complete governance problem.
Organizations still need to decide who owns an agent, what authority it has, which tools it should use, how delegated-user context is represented, and when changes require review.
AI gateways are therefore expanding beyond model routing, but they remain primarily runtime infrastructure.
What are examples of AI gateways in practice?
The role of an AI gateway becomes clearer when several capabilities appear in the same request path.
Example 1: Multi-model customer support application
Suppose a company operates an AI customer-support application that can use several model providers.
Without a gateway, the application may need direct provider credentials and separate integrations for every backend.
With a gateway:
- The application authenticates once to the gateway.
- It sends a request through a common model API.
- The gateway checks the application’s token quota and access policy.
- It resolves the logical model to eligible providers or deployments.
- It selects an approved backend.
- It supplies the provider credential and sends the request.
- If an approved fallback is needed, it routes according to policy.
- It records provider, model, tokens, latency, errors, and estimated cost.
The gateway centralizes provider abstraction, model access, routing, and runtime policy. The customer-support application remains responsible for deciding when and why to call the model.
Example 2: Enterprise support agent using MCP
Suppose an internal support agent can look up customer information, inspect support tickets, and request account changes through MCP.
The agent connects to an MCP gateway instead of directly to every MCP server.
When the agent starts a task, the gateway can return only the MCP tools available to that agent and acting user.
If the agent invokes get_customer, the gateway performs tool authorization and obtains the appropriate downstream credential.
If the agent later invokes delete_customer, the gateway can make a separate authorization decision for that higher-risk action.
The gateway records both decisions along with the MCP server, tool, identity context, latency, and outcome.
The downstream system still enforces the permissions represented by the supplied credential, and the agent runtime still determines whether invoking the tool makes sense for the user’s request.
The gateway controls access to the action. It does not become the application, identity provider, or system of record.
How does Arcade.dev support MCP gateway use cases?
Arcade is the actions runtime for enterprise AI agents. Arcade provides MCP Gateways for controlling how agents and other MCP clients access tools.
An Arcade MCP Gateway presents selected tools from multiple MCP servers through one gateway. This gives teams a governed path for exposing the tools an agent should be able to discover and invoke instead of requiring the agent to connect independently to every underlying server.
Arcade also applies authentication and authorization around tool use, including scenarios where an agent acts on behalf of an end user. Authorization is evaluated per action, at the intersection of what the agent and the user are each allowed to do, so agents don’t need service accounts. Credentials stay outside model context, and access decisions are applied at the action layer before a tool executes. Every invocation is logged with the agent, the user, and the policy decisions applied, and those logs stream to your security information and event management (SIEM) tool through OpenTelemetry.
The gateway corresponds to the MCP and tool-facing portion of the broader AI gateway architecture described in this article. The runtime behind it adds per-action enforcement, centralized governance, and audit. Model gatewaying and provider abstraction for model APIs remain separate concerns.
For a deployment walkthrough covering identity, gateway configuration, and governance controls, see the Arcade operator quickstart.
FAQ
An API gateway manages access, routing, security, and observability for general API traffic. An AI gateway applies similar infrastructure patterns to AI-specific traffic and can add model-provider abstraction, token-aware limits, model routing, MCP server and tool controls, AI-specific telemetry, and other controls that depend on understanding AI resources and protocols.
An LLM gateway or model gateway primarily mediates access to model providers. It typically handles provider abstraction, model routing, credentials, token limits, retries, fallback, and model telemetry. AI gateway can be used as a synonym for this model-facing layer, but the broader category can also include MCP gateways, tool access, and agent-to-agent traffic.
An MCP gateway focuses on traffic between MCP clients and MCP servers or tools, including MCP server access, tool discovery, per-tool authorization, credential brokering, and MCP observability. An AI gateway is the broader category and can include both model and MCP gateway functions together with shared identity, policy, administration, telemetry, and cost controls.
An AI gateway becomes useful when several applications, agents, model providers, or MCP servers need common access controls, credentials, routing, usage limits, reliability policies, observability, or cost allocation. A direct connection can remain simpler for a small application that does not need those shared controls.
Yes, an AI gateway can add latency because authentication, policy checks, routing, credential retrieval, logging, and content inspection happen in the request path. The impact depends on the controls being applied and the gateway architecture. Distributed data planes, caching, efficient policy evaluation, and streaming-aware controls can reduce the overhead.
An AI gateway can reduce risk by limiting MCP server and tool access, detecting changes to governed tool definitions, inspecting visible content, protecting downstream credentials, and recording what happened. It cannot guarantee that every piece of natural-language content or every formally authorized action is safe. Protection still depends on the agent runtime, model safeguards, identity system, tool implementation, and downstream resource.
It can help reduce or control some costs through provider routing, quotas, token-aware limits, caching, usage attribution, and visibility into expensive workloads. It does not automatically reduce total AI operating cost, and it cannot see every cost involved in a task, such as compute, retrieval, storage, browser use, or external SaaS calls.
No. An organization can use one gateway for several AI traffic types or run separate gateways for models, MCP tools, and agent-to-agent interactions. Those components can still share identity infrastructure, policies, configuration, registries, and telemetry.
An AI gateway is primarily a runtime enforcement layer for traffic between AI applications or agents and the resources they use. An AI governance platform operates across a broader lifecycle and can manage inventory, risk, approvals, policies, evidence, and oversight across AI systems. A governance platform can use gateways as runtime enforcement points, but the two categories are not interchangeable.