What is Gateway / router?
By Heemang Parmar · Updated August 2026 · Editorial policy
A gateway or router is an abstraction layer that sits between an application and its AI model providers, routing each request to an appropriate model based on task type, cost, latency, and availability.
A gateway is not a model provider itself; it is the traffic director in front of them. Given a request, it evaluates which provider or model is best suited for this particular job, applies any routing rules, and forwards the call. The application calls one API, and the gateway handles which underlying provider actually answers.
The practical value is flexibility. A gateway that speaks OpenAI-compatible, Anthropic, and Google APIs can route a reasoning task to Claude, a fast classification job to a cheap model, and a vision request to GPT-4o without the application writing separate integration code for each. When one provider has an outage, traffic shifts automatically.
The distinction that matters: a gateway routes; a model developer builds; an inference provider runs the infrastructure. Some products bundle all three roles, which is why the vocabulary gets muddled. Pure-play gateways such as Portkey, Helicone, and OpenRouter provide routing without hosting their own inference.
Why does gateway / router matter?
Gateways matter because model choice should be a configuration, not a hard-coded decision. Routing by task type lets you use a frontier model for complex reasoning and a small fast model for classification, which can cut inference costs dramatically without degrading output quality. Automatic failover also matters: a gateway that shifts traffic when a provider goes down prevents a single outage from taking your product offline.
For teams building on multiple model providers, a gateway also provides observability in one place: token counts, latency, error rates, and spend across every provider in a single dashboard, rather than stitching together insights from separate provider consoles.
How does gateway / router work?
- 1Configure provider credentials: Register API keys for each model provider, and the gateway holds them securely without exposing them to the application.
- 2Define routing rules: Set criteria for how requests get directed: by task type, model family, cost ceiling, latency budget, or fallback order.
- 3Send requests to the gateway: The application calls the gateway's single endpoint using a standard format such as OpenAI-compatible chat completions.
- 4Gateway forwards and augments: The gateway translates the request, routes it to the selected provider, and can attach metadata such as tracing tokens or retry logic.
- 5Response returns through the gateway: The model's response flows back through the gateway, which may attach usage data, cost attribution, and observability headers.
Gateway vs inference provider vs model developer: what does each role do?
| Role | What it provides | Example |
|---|---|---|
| Gateway or router | Routes requests across providers based on rules | Portkey, Helicone, OpenRouter |
| Inference provider | Runs the model infrastructure and serves completions | Anthropic API, OpenAI API, Google AI |
| Model developer | Trains and owns the model weights | Anthropic, OpenAI, Meta |
How is gateway / router used in practice?
Multi-provider routing in one place
ProductOS routes across Anthropic, OpenAI, and Google models for different jobs. A routing layer handles which provider handles each request without the application needing separate integration branches for each.
Reliability through automatic failover
When one model provider has degraded availability, a gateway can reroute traffic to a functioning alternative, keeping agent work uninterrupted rather than failing outright.
Observability across providers
With multiple providers in use, a gateway provides centralized token counts, error rates, and latency breakdowns that a single-provider dashboard cannot offer.
See how Gateway / router works inside ProductOS, from research to shipped code.
Try ProductOS freeFrequently asked questions
What is the difference between a gateway and an inference provider?
An inference provider runs the actual model and serves completions; a gateway sits in front of one or more providers and decides which one handles each request. Some products such as OpenAI and Anthropic effectively bundle both roles: they develop the model, serve the inference, and expose an API that acts as its own gateway. A dedicated gateway product routes across multiple providers without hosting any inference itself.
Is a gateway the same as an API proxy?
A basic API proxy just forwards requests; a smart gateway adds routing logic, cost controls, automatic retries, and observability. The routing intelligence is what separates a gateway from a passthrough.
Do I need a gateway if I only use one model provider?
Not immediately. A single provider with a clean API is fine to call directly. A gateway becomes worth it when you want automatic failover, task-based routing to different models, centralized cost tracking across multiple providers, or the ability to switch providers without changing application code.
Does a gateway add latency?
A thin gateway adds negligible latency, typically single-digit milliseconds. The routing logic is a fast decision, and the request path is the same length as calling the provider directly. Latency becomes a concern only with gateways that do heavy processing such as response transformation or multiple provider calls.
Related terms
- Inference providerAn inference provider is a company that hosts AI models on its own GPU infrastructure and exposes them through an API, handling the scaling, availability, and billing so your application can call completions without managing servers.
- ProviderA provider is a company that makes AI models accessible via an API, operating the infrastructure and managing the commercial relationship so applications can call completions without owning model weights or GPU servers.
- ModelA model is a trained AI system that predicts and generates text, code, or other content based on patterns learned during training, exposed through an API as a component that applications call to produce completions.
- InferenceInference is the process of running a request through a trained AI model to produce an output, covering the full path from tokenizing the input prompt through generating and returning the completion.
- Model developerA model developer is an organization that designs, trains, and owns the weights of an AI model, defining its capabilities, safety behavior, and the core intelligence that inference providers then serve.