What is an LLM? How language models work and where they fail

What makes a language model ‘large’?

An LLM learns statistical regularities in language from large training collections. The label ‘large’ points broadly to its scale of training and capacity, not a single fixed parameter threshold. Depending on its objective, it may generate new text, categorise passages, extract information or transform content. A chat application can wrap an LLM with conversation history, search, files and tools; those components do not magically become properties of the model itself. [2]

The distinction is useful when evaluating a proposed AI feature. A model's ability to compose a plausible answer should not be confused with a guarantee that an application has retrieved the latest information or verified the claim. Explore the wider AI and LLM guide collection for related concepts.

Tokens, embeddings and the input pipeline

Text reaches a language model through a tokenizer. A token may be a complete word, a word fragment, punctuation or another recurring unit. Token boundaries vary by vocabulary and language, so estimating costs or context length by counting words alone is unreliable. Token IDs are converted into numerical representations that network layers can process. [2]

Consider a support message containing a product code, a timestamp and a short explanation. Different tokenizers may split the identifier differently, although a human sees one word. The model operates on tokenised context, not a direct lookup of the sender's intentions. This matters for unfamiliar names, formatting and the length of material supplied.

Attention and the Transformer: the useful mental model

The Transformer architecture introduced in 2017 replaced earlier sequence-processing arrangements with layers built around attention. Attention lets a model weight relationships among positions in a sequence. For autoregressive text generation, causal masking restricts a position to earlier tokens; other model families use different attention arrangements. Attention weights are internal signals, not straightforward proof of how a particular answer was reached. [1] [3]

Not all Transformer-based language models are identical. Encoder-only, decoder-only and encoder–decoder designs support different learning objectives. Masked language modelling and next-token prediction should not be merged into one supposed universal training method. [4]

Training changes weights; prompting usually does not

During pre-training, a model repeatedly predicts training targets, measures error and adjusts parameters through optimisation. The target could be the following token or missing tokens, according to the architecture and learning setup. Dataset preparation, sampling and evaluation strongly influence the system that results. Training occurs before ordinary application use rather than during each conversational reply. [3] [4]

Additional tuning can adapt a pre-trained model to instructions or a specialised task. Fine-tuning changes the model's weights; supplying examples inside a prompt or attaching a document normally changes only the current input context. Distillation is a separate technique for transferring useful behaviour into a smaller model, with its own trade-offs. [5]

How an LLM produces an answer at inference time

A typical request moves through an application, a tokenizer and the trained network. The model produces scores for candidate next tokens, a decoding policy selects one, and the result becomes part of the input for the next prediction. Repetition produces a sentence or longer response until a stopping rule applies. Inference uses learned parameters rather than running the full training procedure again. [2] [3]

Imagine a fictional incident handover: the input contains a short log and the request ‘give the issue, action taken and what still needs checking’. A model may draft the three fields, but the operator must compare them with the original log. This is a conceptual example, not a completed experiment or a measured result.

• Set the task and decide which evidence may be supplied.
• Package instructions and evidence into the model's input.
• Generate tokens using a configured decoding process.
• Compare material assertions with the original evidence.

Context window versus model knowledge versus stored memory

The context window limits how many tokens a system can put in front of a model during inference. Training weights hold learned patterns, not a searchable archive of cited documents. Conversation history may be included in the context; material excluded from the request cannot reliably be assumed to remain accessible. A longer context is helpful for some tasks, but does not by itself guarantee faithful retrieval or a correct conclusion. [2] [3]

External search, document retrieval and tool use require application components outside the raw language model. A retrieval pipeline can provide current passages but the output must still be checked against them. Our guide to the Model Context Protocol covers the boundary between a model, an application and connected tools.

Where language models can help—and what to validate

Good candidates include drafting a first summary, classifying inbound requests, extracting fields from narrative text, rephrasing technical documentation and assisting with code. Success should mean a useful outcome for the specific task, not merely a readable answer. Format compliance, missing details, incorrect claims and the time needed for human corrections all matter.

For extraction, define a schema and a rule for unknown fields. For summarisation, preserve source material and forbid facts that are not in it. For code, retain normal tests, threat modelling and review. No model response substitutes for checks required by the surrounding workflow.

Hallucinations, privacy and prompt injection

Generated statements can be confidently wrong. Training data may reflect biases or gaps. A quoted document may contain instructions designed to redirect the model, and a connected system can expose confidential information or create unwanted side effects if permissions are too broad. NIST's generative AI risk profile highlights information integrity, security, privacy and over-reliance among risks requiring active management. [6]

A large parameter count, long context window or attractive benchmark score is not proof that a system is safe for a particular use. Any tool-enabled application should apply access control outside the model, treat retrieved text as untrusted and require confirmation for high-impact operations. See the MCP security checklist for practical controls.

A model is not the same thing as an agent

A model generates or processes tokens. An agent-like application may also plan steps, call external tools, read resources, check outcomes and decide whether to continue. That sequence is constructed by software around the model, with explicit authentication and authorisation. A fluent description of an action is not evidence that the action took place.

For example, answering ‘the calendar is updated’ is only a sentence unless an authorised calendar operation actually succeeded and its result was checked. Keeping the distinction visible helps prevent invented success claims and accidental tool access. Our MCP explainer focuses on the connection layer, not the capabilities of an LLM in isolation.

A practical checklist for choosing an LLM

Start from a small but representative set of real tasks, written acceptance criteria and examples of unacceptable errors. Compare outputs with reference answers, record omissions and factual mistakes, and check whether the result follows the intended structure. Note the model version and settings. Without a controlled procedure, anecdotal comparisons say little about reliability.

Then account for where data goes, the cost of inference, response time, operational maintenance and the human effort left over. The AI tool evaluation worksheet offers a repeatable approach. For deployment trade-offs, see local AI versus cloud AI.

• Define the task and allowed source material.
• Choose acceptance criteria and must-not-fail conditions.
• Include ambiguous, incomplete and hostile input cases.
• Verify important facts and the visibility of evidence.
• Set an explicit hand-off to a human reviewer where necessary.