If you've made much use of LLMs, even very good ones, you have likely encountered their propensity for mistakes. Weird mistakes, too, given how capable LLMs often are. The best LLMs today can complete complex software engineering tasks in minutes, while sometimes attempting to send an email to an address ending in @example.com, seemingly believing it was real [1]. Only sometimes, though, which could strike you as even stranger. You might send the exact same prompt to the exact same model and yet still wind up with two different answers, only one of which is actually correct. What causes this? Why is it that these models which appear so capable on tasks the average person wouldn't even know where to start on can also abruptly crumble when given problems a child could solve without even thinking about it [2]? How can we trust them to do anything properly if they can make such elementary mistakes?
Hallucinations
The term the industry has settled on for the strange behaviour in which an LLM generates a sentence with a clear factual error that a human with access to the information probably wouldn't make is hallucination. Common examples of hallucinations might be citing a legal case or a book which simply does not exist (or if it does, does not say what the LLM claims), writing code that depends on a misnamed function or variable, or making claims which are objectively untrue.
Note that hallucination can be vaguely defined sometimes; it's not a very rigorous term. For our purposes, I'll say that a hallucination is a factual error produced by an LLM which lacks the markers that usually indicate uncertainty. Sometimes people will say that this is the model "confidently making up garbage" or even "lying".
An example of a hallucination might look like this:
USER: Who was involved in the issue of the Magna Carta?
ASSISTANT: The issuing of the Magna Carta in 1215 involved King John of England, Stephen Hawking, William Marshal, Henry de Loundres, and Hugh de Neville, among others.
Stephen Hawking was not, in fact, present for the sealing of a document which occurred 727 years before his birth, though the other people named were real and involved. "Stephen Hawking" was a real person, was from England, and a Stephen was actually involved in the real document (Stephen Langton, the Archbishop of Canterbury). Yet no one who knows even a little bit about Hawking and the Magna Carta could make such a mistake. Note that this example is not one that I have personally encountered in an LLM. It is simply a plausible illustrative example of such an error.
Importantly, if you did not read the passage closely or did not know who Stephen Hawking was or what the Magna Carta is, you would have little way of telling that something was wrong. It is embedded within a correct list of names. Yet it is simply wrong. This is why people might refer to the model as being "confident" in its incorrectness, although the text itself doesn't tell us much about the model's confidence [3].
Why Hallucinations Happen
If the above case were real, we could make some guesses about how it came about. In a vacuum, if you were trying to predict the word which came after "Stephen", then "Hawking" would probably be pretty high on the list, along with "King" and "Spielberg". "Langton" would have quite a low probability, except that the other names (King John, William Marshal, etc.) and the presence of the words "Magna Carta" should push the distribution much further towards "Langton" than the other surnames.
But "more likely" does not mean "guaranteed". Recall that what models really give you is a list of all possible tokens and their probabilities (a probability distribution), from which we pick one at random, weighted by the probabilities the model predicted. In a true probability distribution, no token would ever have a probability of zero, meaning that in principle, many might be selected. In practice, most have values so close to zero that they would get rounded away, but in this particular case, it is fairly likely that the other plausible surnames for Stephen had meaningful chances, so "Hawking" might have come up by pure chance, even if "Langton" were more likely. This would be a case in which the non-deterministic behaviour of token selection worked to produce a false statement.
Alternatively, the model itself may simply have produced a probability distribution that does not line up with real facts about the world. To an LLM, the text it sees during training is the "ground truth" to which it naturally has access, but from our perspective there is nothing to ensure the model's ground truth matches ours. Maybe information about "Langton" didn't end up in its training data, or the model incorrectly learned that "Hawking" probably follows "Stephen" under any circumstances.
In this case, the hallucination would be of little consequence beyond teaching someone something wrong about a minor historical character. In higher-stakes cases, there could be more severe consequences [4], so minimising such mistakes is naturally ideal. The deeper problem when it comes to LLM hallucinations is that, as far as the model is concerned, it is not straightforward to tell the difference between "hallucinations" and standard operating procedure. In either case, the model generates a probability distribution over next-tokens and we sample from the distribution. If the model or the random selection selects a token that results in an incorrect statement, it has no way to go back and fix it.
I want to be clear that this is not me saying that LLMs routinely make up facts. In recent years, LLM trainers have substantially reduced the incidence of hallucinations and made LLMs more inclined to search for and link to external information when doing so would be helpful. They do not want their models making things up for the obvious reason that you would be less willing to use them if they often did so.
Furthermore, we don't call every failure of an LLM a "hallucination", even if the mechanism is the same. Especially for domains like coding or maths, a model getting a wrong answer or code that does not work is not usually considered a "hallucination"; it just wrote bad code or an incorrect proof. Yet what the model is doing when it "hallucinates" is, in fact, mostly the same as when it is simply wrong, or when it is actually right.
Correctness is Bigger Than Tokens
The issue is that if you want to answer a question or write some code, the correctness of the answer or usefulness of the code is not contained in just one token. Rather, it's contained in many tokens, strung together. This could be hundreds or sometimes even hundreds of thousands of tokens, depending on the question.
When we use an LLM to generate answers or code, we bring our own standard of "correctness" to the model (what the truth of the matter actually is or what the computer will do when the code is run, for example). An LLM lacks an internal measure of truth beyond how it has learned to be "not-wrong" about the very next token in the sequence. There is no equation that lets you put in any statement created using language and gives you a "true" or "false" in return [5].
This is a challenge if we want to train our LLMs to be factually accurate. The training objective is "most likely next token", and that is not the same question as "next token which would contribute to a true statement". We would hope that our training data is mostly composed of true statements — it's a vote of extreme confidence in the wisdom of the crowds that we use data from the internet in our training corpora at all — but we know this is not always going to be the case. Fiction, disagreeing perspectives, obsolete or debunked theories, opinions, and questions without answers are a core part of human knowledge, so they inevitably wind up as part of the training data for an LLM [7].
We must, therefore, evaluate a model's tendency to produce correct or true text from the outside. For a given task, we have to provide an external judge which can assess whether the text generated by a model fits our criteria.
Benchmarks: Judging from the Outside
At a small scale, a person could simply use an LLM for the tasks they care about and evaluate whether or not it works properly based on their own criteria. However, this is not a very comprehensive approach (maybe you only have time to evaluate a couple of specific tasks) and we also want to make sure our comparisons between models are fair, using the same set of problems for each one.
The industry does this using benchmarks. These are essentially exams that we make models sit to evaluate their abilities in different areas that we care about. Much like exams you might hand to students, a benchmark is a list of questions for which we have specific answers or types of answers in mind. Unlike exams handed to students, though, benchmarks are often run several times on the same model in order to smooth out the randomness inherent to LLM usage. Since benchmarking is done "at inference-time", when the weights are frozen, the model is unable to learn anything about the exam between runs, so this is not an unfair part of testing as it would be if schools ran the same exam multiple times on the same students.
Some benchmarks will include multiple-choice questions with specific answers in mind. Others will allow the model to generate freeform text and then grade that text based on certain criteria (this could be done by a person, but these days it is quite common for other LLMs to be used as judges for the sake of convenience). Still others might give the model a more complex environment with various tools it is allowed to use to solve a problem and then use a special "submit answer" tool to indicate that the model thinks it is done.
When Benchmarks Go Wrong
Unfortunately, benchmarks themselves are fallible measures. There are a few ways a benchmark score might misrepresent a model's true capabilities, and all must be kept in mind when weighing one up.
Benchmark Contamination
For one, benchmarks should be comprised of questions that have never been included in the model's training data and thus have never been used to update the model's weights. If a benchmark question were in the training data for an LLM, then it would have been used to update the model's weights, which unfairly biases the model towards answering the question correctly. Just as we want a child at school to understand the underlying patterns in arithmetic, rather than just the answer to a specific question, we also want an LLM to have learned the underlying facts we test it on. When a model is trained on the same questions used to benchmark it, we call this benchmark contamination (sometimes it is also called data leakage), and it makes the test invalid.
Faulty Benchmarks
An ideal benchmark will be large (so that we know it wasn't just luck that the LLM got the answer right), diverse (so that we're testing more than the exact same behaviour over and over), and correct. That last one might seem obvious — clearly an exam's answer sheet should have the right answers on it — but this is surprisingly difficult for very large benchmarks, since this requires a person or team to go through it meticulously, which is expensive [8]. Correctness has actually been a problem for a few benchmarks in the past, which have been found to contain errors [9], meaning that a low score (or a high one) may not indicate a model's true abilities.
Benchmarketing
Benchmarks are used by LLM trainers as a way of determining if their models are working or not (and thus whether they are ready for release), and are also often used to advertise the quality of the model before release. Especially in today's highly competitive LLM landscape, the labs that train them will typically announce and release their new models with a series of benchmark scores and comparisons to other models (both the lab's older models and those of their competitors) on the same set of benchmarks. This will be something like OpenAI claiming that their new GPT has a score of 71 on CodeThatWorksGoodBench, compared to Claude's 68, or Alibaba saying that the new Qwen gained an extra 15 points on FrontierCooking compared to the one they released three months ago.
This is all fairly standard marketing fare. As new LLMs come out, they will typically jostle for the top spot. Cost is also a major consideration which can offset a lower benchmark score. Depending on your use case, you may prefer a model that is "good enough" at a lower cost than the one scoring the highest on a given benchmark. It is also generally true that labs will release their models with the benchmarks that make them look like the best value. Sometimes this is simply being selective about which benchmarks you show, though sometimes it can be a little more nefarious.
In some cases, a model's benchmark scores are not reflective of their real ability (usually for the worse). When done intentionally, this is sometimes called benchmaxxing, which can involve training the model on questions that look suspiciously similar to those on a popular benchmark, data leakage (this is not always intentional if the dataset is on the public internet, but is still a bad look), or cherry-picking specific testing runs in which the model happened to perform best. As with all marketing, it is best to take vendor-published numbers with a grain of salt and try to find trustworthy, independent benchmarks for your particular use case.
Trust, But Verify
At the end of the day, evaluating the truthfulness or usefulness of LLMs is not a straightforward task and ensuring it is harder still. We do not always have certainty about the quality of a text an LLM has produced. We can do our best with benchmarks and tests of various kinds, but these too can be flawed, so they should be taken only as evidence that an LLM is working to our satisfaction, rather than as proof.
"Which token comes next?" and "Which token would make this statement true?" are simply not the same question, though we can do our best to make their answers overlap. LLMs will always have the opportunity to produce false-but-convincing output, conflations, contradictions, and mistakes. The most practical response to this is to be careful. Sometimes, the consequences of a mistake are small, so we don't need to worry too much. Sometimes they are not, and our caution when using LLMs should reflect this. Ideally, it is best to constrain LLMs where possible and to provide them with meaningful ways of checking their work. We will discuss exactly how such information is delivered and what we can do with it in the next article in which we'll explore the Anatomy of a Context Window.
[1] https://arxiv.org/html/2606.13715v2
[2] A fun example of this that floated around is the "car-wash problem". Ask a model "The car wash is 50 metres from my house. Should I walk or drive there?" and, for a while at least, nearly all of them would tell you to walk because it's such a short distance. https://mastodon.world/@knowmadd/116072773118828295
[3] "Confidence" means something different for LLMs than it does for people. The best measure we can get of an LLM's confidence about a given token is how strongly a given token was weighted compared to alternatives. The trouble with this is that it bundles up a number of reasons the next token might be uncertain beyond just that the model does not know the fact. For instance, if there are several different ways of saying the same thing, the model might not give any one of the starting tokens for these phrases a decisive advantage over the others, even if it is "confident" about the underlying fact all of them are describing.
[5] For some domains, like maths, this is possible. But how would you determine the single correct answer to the question "What is the best way to eat potatoes?" [6]
[6] It's mashed, nice and creamy, with loads of garlic, btw
[7] Further still, if we accept that false information should be scrubbed from an LLM's training corpus, we make those who compile the corpus the arbiters of what is true to the model. I hope that most people, even those who broadly trust and agree with the LLM trainers, would see that this is not wise.
[8] And, I say as someone who has needed to manually read and label large datasets of code for security vulnerabilities, unbelievably, soul-destroyingly boring.
[9] https://aclanthology.org/2025.naacl-long.262.pdf, https://gradientscience.org/gsm8k-platinum, https://github.com/sunblaze-ucb/exploitgym/blob/main/CHANGELOG.md