What is Generation / completion?
By Heemang Parmar · Updated August 2026 · Editorial policy
Generation or completion is the output an AI model produces in response to a prompt, built token by token from the most statistically plausible next token given everything in the context window.
The model produces text one token at a time, and each token is chosen based on probability distributions learned during training. This is why the same prompt can give different answers on different runs: the model samples from its distribution rather than picking one fixed next token.
Generation is the core operation everything else builds on. A chat response, a code file, a translated paragraph, a summarized document: all are completions generated from an input prompt. The quality of a completion depends on the model, the prompt, the context, and the sampling settings that control how deterministic or creative the output is.
The practical distinction is input versus output. Tokens sent to the model are input; tokens returned are the completion or generation. Most API pricing charges these separately, with output tokens typically costing several times more per token because they require sequential generation rather than parallel processing.
Why does generation / completion matter?
Generation matters because it is the unit of cost and latency in AI products. Input tokens pay once per request; output tokens pay per token generated, which for long-form responses can be the dominant cost. Understanding what drives completion length and how to constrain it, via response format specifications or output token limits, directly affects both the quality of results and the invoice at the end of the month.
For product teams, the completion is also where quality is verified. Context curation, prompt design, and model selection all exist to produce better completions, and evaluation frameworks measure whether the output meets the bar rather than how cleverly the request was constructed.
How does generation / completion work?
- 1Tokenize the prompt: The input is split into tokens, mapped to numeric IDs, and loaded into the model's context window.
- 2Predict the next token: The model computes probability distributions over its vocabulary for what comes next, conditioned on the full context.
- 3Sample from the distribution: The model selects the next token, with temperature and top-p settings controlling how deterministic or varied the selection is.
- 4Append and repeat: The chosen token gets appended to the sequence, and the model repeats until it produces a stop token, hits a length limit, or decides generation is complete.
Generation vs inference vs completion: are they the same thing?
| Term | What it describes | Scope |
|---|---|---|
| Generation | The act of producing output tokens | The output itself |
| Completion | The finished output text | The result, synonym for generation |
| Inference | The entire process of running a request through a model | Input tokenization plus generation |
How is generation / completion used in practice?
Code generation with the Fullstack Builder
ProductOS's Fullstack Builder generates code in a live sandbox, producing multi-file completions that implement features described in plain language. The generated code syncs to your own GitHub repository.
Long-form document generation
The PRD Agent generates complete product requirement documents from a described idea, producing structured completions that follow Amazon PRFAQ, Lean, or Opportune templates.
Structured output generation
Through system prompts and response format specifications, models generate structured JSON that downstream application code parses, turning free-form generation into reliable API responses.
See how Generation / completion works inside ProductOS, from research to shipped code.
Try ProductOS freeFrequently asked questions
What is the difference between generation and inference?
Inference is the full process: tokenizing the input, running it through the model, and producing output. Generation or completion specifically refers to the output tokens themselves. When someone says a request consumed 500 tokens of inference, they usually mean 300 input tokens and 200 generated tokens.
Why does the same prompt give different answers each time?
Models do not return one fixed answer; they sample from a probability distribution for each token. Temperature settings control how peaked or flat that distribution is. A temperature of zero almost always picks the highest-probability token; higher temperatures introduce variation. For consistent output, lower the temperature and specify the format explicitly.
Why are output tokens more expensive than input tokens?
Input tokens are processed in parallel across all positions in the context window. Output tokens are generated sequentially, with each one requiring a full forward pass through the model, and that sequential dependency makes generation the computationally expensive half of inference. Providers price accordingly.
What is token limiting and why does it matter?
Most APIs enforce a maximum number of output tokens per request. If that limit is reached, the completion stops abruptly. Setting the right output limit involves knowing roughly how long the expected answer should be, since too short a limit truncates useful output and too long wastes inference budget on unused capacity.
Related terms
- InferenceInference is the process of running a request through a trained AI model to produce an output, covering the full path from tokenizing the input prompt through generating and returning the completion.
- TokenA token is the basic unit of text that AI models read and generate, roughly four characters or three-quarters of an English word; model pricing, context windows, and generation speed are all measured in tokens.
- Context windowA context window is the maximum amount of text, measured in tokens, that an AI model can process in a single request, covering the system prompt, conversation history, documents, and the model's own response.
- PromptA prompt is the instruction given to an AI model that specifies the task, context, constraints, and output format the model should follow, forming the entire interface between your intent and the model's response.
- Large language model (LLM)A large language model (LLM) is an AI model trained on massive text datasets to predict and generate language, powering writing, coding, analysis, and reasoning tools through token-by-token text generation.