What an LLM actually does when it answers a question, why it sometimes gets things wrong, and which tasks it is well suited to.
The short version
A large language model (LLM) is a program trained to predict the next piece of text given the text before it. By repeating that prediction many times, it can write answers, summaries, code and translations. Everything else, from chat assistants to AI agents, is built on top of this one capability.
From text to tokens
Models do not read words or letters directly. Text is first split into tokens, which are short chunks such as a whole word, part of a word or a punctuation mark. Each token is mapped to a number the model can process. The model reads a sequence of tokens and produces a probability for every possible next token.
What the transformer does
Most current LLMs use the transformer architecture introduced in 2017. Its key mechanism, attention, lets every token weigh how relevant every other token in the input is. That is how the model connects a pronoun to the noun it refers to, or a question to the relevant sentence several paragraphs earlier.
Training: pre-training and tuning
Pre-training exposes the model to a very large body of text so it learns grammar, facts and patterns by predicting missing or next tokens. The base model is then tuned with instruction examples and human or AI feedback so it follows requests, uses a helpful tone and declines unsafe tasks. Neither step gives the model a database it can look facts up in.
Inference: generating an answer
When you send a prompt, the model predicts one token, appends it, and repeats until it reaches a stop condition. Settings such as temperature control how often it picks less likely tokens. Lower values give more consistent output; higher values give more variety. Each generated token costs compute, which is why long answers take longer and cost more.
Why LLMs make mistakes
Because the model generates plausible text rather than retrieving verified facts, it can produce confident statements that are wrong, often called hallucinations. It also has a training cut-off, so it does not know about recent events unless you provide that information. It can struggle with exact arithmetic, long chains of logic and information buried in very long inputs.
Where LLMs work well
LLMs are strong at language tasks: summarising, drafting, rewriting, classifying, extracting fields from documents, translating and explaining code. They are weaker as a sole source of truth. Most reliable business applications pair the model with trusted data (for example through retrieval), clear instructions, and checks on the output.
Choosing a model
There is no single best model. The right choice depends on task difficulty, acceptable latency, cost per request, context length, data privacy requirements, and whether you need to run it on your own infrastructure. Test two or three candidates on your own examples rather than relying on public leaderboards alone.
Key takeaways
- An LLM predicts the next token; everything else is built on that.
- It generates plausible text, not verified facts, so ground it in trusted data for factual tasks.
- Model choice is a trade-off between quality, cost, latency, privacy and context length.
- Evaluate candidates on your own examples.
If you are planning a product feature built on an LLM, our Generative AI team can help you choose a model and design the system around it.
Generative AI Development- Attention Is All You NeedVaswani et al., Google · 2017 · Research paperarxiv.org
- Training language models to follow instructions with human feedbackOuyang et al., OpenAI · 2022 · Research paperarxiv.org
- LLM CourseHugging Face · Coursehuggingface.co