What is Deployment / endpoint?
By Heemang Parmar · Updated August 2026 · Editorial policy
A deployment or endpoint is a hosted, callable instance of an AI model accessible via an API, where the model runs on the inference provider's infrastructure and receives requests from your application.
When you call the Anthropic API or the OpenAI API, you are calling a deployment. The model weights are loaded on servers the provider manages, and your request travels over HTTPS to an endpoint URL that routes it to an available instance. You do not manage the infrastructure; you send prompts and receive completions.
The key distinction from self-hosting is ownership of the hardware. A hosted deployment means the inference provider handles capacity planning, GPU allocation, model updates, and uptime. You pay per token or per call. A self-deployed model means you own the machines, the networking, the scaling logic, and the updates.
Each deployment has an endpoint URL and an API key that authenticates your requests. Multiple deployments can exist for the same model: one for development, one for production, each with its own key, rate limits, and spending controls. Monitoring which deployment a request hit is usually the first diagnostic step when investigating latency or quality issues.
Why does deployment / endpoint matter?
Deployments matter because they are the unit of availability and cost. A model is only as accessible as its deployment; if the endpoint is down or rate-limited, your application cannot generate completions regardless of how good the model is. Understanding which deployment backs your API calls, and whether the provider routes across multiple instances, is essential for setting reliable expectations for your users.
For cost management, deployments are where you set spending caps, monitor token usage, and decide which requests go to which model. A production deployment might route to Claude 3.5 Sonnet while a batch processing job routes to a smaller model through a different deployment, each with its own budget.
How does deployment / endpoint work?
- 1Provision a deployment: In your provider console, select the model and region, configure authentication, and note the endpoint URL and API key.
- 2Configure access controls: Set rate limits, spending caps, and IP allowlists on the deployment to protect against unexpected spikes.
- 3Send requests to the endpoint: Your application calls the endpoint URL with your API key, the model identifier, and the prompt in the provider's expected format.
- 4Provider routes to an instance: The endpoint receives the request and routes it to an available instance of the specified model running on the provider's GPU infrastructure.
- 5Response returns to your application: The completion flows back over HTTPS to your application, which parses and uses it.
Hosted deployment vs self-deployment vs serverless endpoint: what is the difference?
| Approach | Who manages infrastructure | Best for |
|---|---|---|
| Hosted deployment | The inference provider | Most teams; zero infrastructure management |
| Self-deployment | Your team | Strict data governance, very high volume |
| Serverless endpoint | The provider, auto-scaling | Variable workloads with no capacity planning |
How is deployment / endpoint used in practice?
Agents backed by hosted model deployments
ProductOS agents run on model deployments from Anthropic, OpenAI, and Google, each endpoint handling requests routed by the platform's multi-provider layer. BYOK lets you point those calls at your own deployments for cost attribution.
Staging and production endpoints
Development and production workloads can target different deployments of the same model, each with its own rate limits and monitoring, so testing does not consume production budget.
Region selection for latency and compliance
Providers let you select which geographic region your deployment runs in, which matters for both the latency of responses and the data residency requirements of your users.
See how Deployment / endpoint works inside ProductOS, from research to shipped code.
Try ProductOS freeFrequently asked questions
What is the difference between a deployment and an API endpoint?
A deployment is the running instance of a model on the provider's infrastructure. The endpoint is the HTTPS URL that receives requests to that deployment. In practice the terms are used interchangeably when people say 'call the Anthropic endpoint' to mean 'send a request to the Anthropic API.'
Can I deploy the same model multiple times?
Yes. Multiple deployments of the same model can exist in parallel, each with its own endpoint URL, API key, rate limits, and usage tracking. This is common for separating development from production or for giving different internal teams their own quotas.
What happens when a deployment hits rate limits?
The API returns an error with a 429 status code, and your application should handle it with exponential backoff and retry. For critical production systems, multi-deployment and multi-provider routing provide fallback capacity so a single rate-limited deployment does not cause an outage.
Do model updates replace my deployment?
It depends on the provider's versioning policy. Some providers update the deployment in place to the latest compatible version; others require you to explicitly migrate to a new deployment for a new major version. Understanding the versioning and update policy matters for production stability, since an unexpected model swap mid-deployment can change output quality.
Related terms
- Inference providerAn inference provider is a company that hosts AI models on its own GPU infrastructure and exposes them through an API, handling the scaling, availability, and billing so your application can call completions without managing servers.
- InferenceInference is the process of running a request through a trained AI model to produce an output, covering the full path from tokenizing the input prompt through generating and returning the completion.
- ModelA model is a trained AI system that predicts and generates text, code, or other content based on patterns learned during training, exposed through an API as a component that applications call to produce completions.
- AI agentAn AI agent is a software system that uses a language model to plan and execute multi-step tasks toward a goal, calling tools, checking results, and adjusting its approach without step-by-step human instructions.
- ProviderA provider is a company that makes AI models accessible via an API, operating the infrastructure and managing the commercial relationship so applications can call completions without owning model weights or GPU servers.
- Gateway / routerA gateway or router is an abstraction layer that sits between an application and its AI model providers, routing each request to an appropriate model based on task type, cost, latency, and availability.