What is Inference?
By Heemang Parmar · Updated August 2026 · Editorial policy
Inference is the process of running a request through a trained AI model to produce an output, covering the full path from tokenizing the input prompt through generating and returning the completion.
Inference starts when your application sends a prompt to a model endpoint and ends when the completion is back in your application. In between, the model tokenizes the input, runs it through its neural network layers, generates output tokens one at a time, and streams them back. The cost and latency of inference are what product teams engineer around: batching requests, caching results, routing to cheaper models, and constraining output length all exist to manage inference economics.
The two cost axes are input tokens and output tokens. Input tokens are processed largely in parallel across the model's context window. Output tokens are generated sequentially, each requiring a full forward pass, which makes generation the computational bottleneck. That asymmetry is why output token pricing is typically several times higher than input token pricing.
Inference is distinct from training. Training is the expensive, offline process of adjusting the model's weights using data. Inference is what happens at runtime when the model is already trained. When someone says a model is expensive to run, they mean inference is expensive, which is why model developers and inference providers compete heavily on inference efficiency.
Why does inference matter?
Inference matters because it is where the money goes in AI products. API costs, latency budgets, and availability SLAs are all inference concerns. A product that ignores inference optimization can find itself with excellent AI features and a runaway cost curve. Understanding what drives inference cost per request, and how model choice and prompt design affect token counts, is the operational literacy that makes AI products profitable rather than ones that scale into an invoice problem.
It also matters for reliability. Inference latency determines how responsive your AI feature feels. Inference failures, timeouts, and rate limit errors need to be handled gracefully in your application code. Treating inference as a black box that always returns promptly is how production systems fail visibly when traffic spikes.
How does inference work?
- 1Tokenize the input: The prompt is split into tokens and mapped to numeric IDs the model's embedding layer can process.
- 2Process through the model: The tokenized input passes through the model's layers, with attention mechanisms relating each position to every other.
- 3Generate output tokens: The model produces one token at a time, each choice conditioned on all previous tokens, until it produces a stop token or hits the maximum length.
- 4Return the completion: Generated tokens are assembled into text and returned, usually streamed token by token over HTTPS for lower perceived latency.
Inference vs training vs fine-tuning: what is the difference?
| Process | When it happens | Who pays for it |
|---|---|---|
| Inference | Runtime, when a user makes a request | Per-token API costs |
| Training | Offline, before deployment | Massive GPU compute, developer cost |
| Fine-tuning | Offline, on custom data | Training run plus evaluation |
How is inference used in practice?
Agents that run inference on each step
ProductOS agents run inference on every action: researching, writing the PRD, generating designs, coding, and testing. Each agent step is an inference call, and the platform manages which model handles each call.
Streaming completions for responsiveness
ProductOS agent outputs stream over the connection so progress is visible as the agent works, rather than waiting for the full completion to arrive before displaying anything.
Cost control through token management
By sharing one project context across all stages, ProductOS avoids resending the same background information in every inference call, which directly reduces input token costs across the pipeline.
See how Inference works inside ProductOS, from research to shipped code.
Try ProductOS freeFrequently asked questions
What is the difference between inference and training?
Training adjusts the model's weights using data, which is expensive and happens offline. Inference runs a trained model against new inputs to generate outputs, which is what happens in production every time your application calls the API. Inference is what you pay for per token when you use a hosted model.
Why is inference expensive?
Large models require significant GPU compute to run each inference request, and each output token is generated sequentially rather than in parallel, which is the fundamental speed bottleneck. Providers price to recover the hardware and energy costs plus margin. Competition and model efficiency improvements have driven costs down substantially since 2023, but inference is still the dominant cost in AI products.
How can I reduce inference costs?
Three main levers: reduce input token count by trimming context and summarizing history, route simple tasks to smaller cheaper models, and constrain output token limits to stop generating before the model produces more than you need. Prompt caching, where the provider caches the static parts of your prompt, can also substantially reduce costs for repeated query patterns.
What is streaming in inference?
Streaming returns tokens one by one over the connection as they are generated rather than waiting for the full completion to be ready before returning anything. This reduces perceived latency for the user. Most model APIs support streaming; whether your application does depends on how it handles the response stream.
Related terms
- TokenA token is the basic unit of text that AI models read and generate, roughly four characters or three-quarters of an English word; model pricing, context windows, and generation speed are all measured in tokens.
- Large language model (LLM)A large language model (LLM) is an AI model trained on massive text datasets to predict and generate language, powering writing, coding, analysis, and reasoning tools through token-by-token text generation.
- Deployment / endpointA deployment or endpoint is a hosted, callable instance of an AI model accessible via an API, where the model runs on the inference provider's infrastructure and receives requests from your application.
- Inference providerAn inference provider is a company that hosts AI models on its own GPU infrastructure and exposes them through an API, handling the scaling, availability, and billing so your application can call completions without managing servers.
- Generation / completionGeneration or completion is the output an AI model produces in response to a prompt, built token by token from the most statistically plausible next token given everything in the context window.