How tokens and context windows work, how they drive cost and latency, and practical ways to fit the right information into a prompt.
Why this matters
Almost every practical limit of an LLM application, from cost to response time to answer quality, comes back to tokens. Understanding them helps you design prompts and systems that stay fast and affordable as usage grows.
What a token is
A token is the unit a model reads and writes. In English, a token is often a word or part of a word, so a sentence usually contains more tokens than words. Other languages, code and unusual strings such as IDs can use noticeably more tokens for the same amount of text. Most providers publish a tokenizer so you can count exactly.
What the context window is
The context window is the maximum number of tokens the model can consider in a single request. It includes everything: the system instructions, conversation history, any documents you insert, the user message and the model's own reply. If the total exceeds the limit, the request fails or older content has to be dropped.
How tokens affect cost and latency
Providers typically charge separately for input and output tokens, with output usually priced higher. Longer inputs take longer to process before the first word appears, and longer outputs take longer to finish. A prompt that quietly grows with every conversation turn can multiply costs without anyone noticing.
Long context is not free recall
Context windows have grown large enough to hold entire reports or codebases. Research has shown, however, that models do not always use information evenly across a long input; details in the middle can be missed more often than details at the start or end. More context also means more cost and slower responses, so relevance matters more than volume.
Practical techniques
Keep system prompts focused and remove instructions that no longer apply. Summarise older conversation turns instead of resending them in full. Retrieve only the passages relevant to the current question rather than whole documents. Where your provider supports it, prompt caching can reduce the cost of repeating a long, unchanging prefix. Set a sensible maximum output length for each feature.
Monitor token usage
Log input and output token counts per request and per feature. This makes it easy to spot a prompt that has grown too large, a user flow that triggers unusually long answers, or a feature whose cost is out of line with its value.
Key takeaways
- Everything in a request, including the reply, counts toward the context window.
- Input and output tokens drive both cost and latency.
- A larger context window does not guarantee the model will use every detail.
- Send the most relevant information, not the most information.
If token costs or response times are limiting your AI feature, our team can review the prompt and retrieval design with you.
Generative AI Development- Lost in the Middle: How Language Models Use Long ContextsLiu et al. · 2023 · Research paperarxiv.org
- TokenizerOpenAI · Toolplatform.openai.com
- Prompt cachingAnthropic, Claude Docs · Documentationplatform.claude.com