In my last two posts, I discussed what it means to model language, how we do it, and how it allows us to construct coherent (and even useful) passages of text. Recall that the question we ask of models like ChatGPT and Claude is "Given this text, which token comes next?". What is surprising about LLMs is that answering that question accurately and quickly is the backbone of a multi-trillion-dollar industry, since it seems rather mundane at first glance.
This article will explain why such a simple-sounding task has become so consequential. In short, it is because these models are huge. But just how huge are they? What does it take to make them so big, and why did we decide to? What's so special about making your language model large?
Just How Large?
Firstly, I want to give you some perspective on the size of these things. Typically we measure models by the number of weights they contain (the standard term is parameter count), without necessarily worrying ourselves yet about how those weights are arranged. Going back to our bird classifier from Part 1, some older image recognition models from the early 2010s had parameter counts in the tens of millions [1]. This would come out to a couple of hundred megabytes of storage space — a few minutes of 4K video or a few hours of audio. Today, since we have made a number of improvements to how we train ML models, a modern image classifier might have as few as five million parameters.
These days, the largest LLMs with published parameter counts are typically in the high hundreds of billions to low trillions [2]. "Small" language models (SLMs), which work identically to LLMs but simply have fewer parameters, usually top out at around ten billion parameters [3]. Thus, the language models we see today can be many thousands of times larger than our bird classifier.
A to-scale comparison of parameter counts for four neural networks — AlexNet (60M), MobileNetV3 (5M), Qwen 3.5-2B (2B), and GLM 5.2 (744B) — where each box represents 10 million parameters and the grid scrolls horizontally.
Why Do They Need to Be So Big?
Even without fully knowing exactly what models of these sizes mean in practical terms (which we will discuss later), you might appreciate that it is quite substantial for a large language model to be potentially hundreds of thousands of times larger than our little bird classifier, and it is reasonable to ask why. In short, this is because language is simply much, much more complicated than bird identification.
Realistically, to identify a bird, you only need to look for a narrow, bounded set of patterns. Fiddler crabs, for example, can distinguish between predatory birds and those that are harmlessly passing by [4]. They do not need very complex eyes to do this — when they respond, a distant bird may only be visible to one or two of the facets in their compound eyes, making its shape impossible to make out — yet they are still able to perform the threat detection they need to survive.
Conversely, we humans are the only living creatures who use an open-ended language. Not all of our comparatively enormous brains go into language processing, but their size tells us something about the sheer scale of what language processing requires in nature. Even though artificial and biological neural networks are very different, the scale required for language processing remains enormous in both. Patterns in language are so complex and so open-ended that the odds are good that most of the sentences in this article have never been written before by anyone, yet you are still able to understand what they mean. That is the complexity of language, and that complexity is why LLMs are so large.
While a smaller language model might learn common sentence structures, very well-known facts, or the content of influential literature, at a certain point it becomes increasingly difficult for new information to be represented in the model's weights without interfering with something else (and potentially something important). A model with only a few dozen parameters obviously couldn't learn much that's useful — virtually nothing in the grand scheme of language — but the same limitation applies even to models with billions of parameters. There might be highly uncommon patterns that have not appeared frequently enough in language to be learned, such as how to write a program that does something unusual and specific in an uncommon programming language. Other patterns might be so deep and complex that they require mountains of background knowledge to make accurate next-token predictions about, such as the solutions to long-standing mathematical problems.
As the number of parameters and the amount of text we train our models on go up, the complexity of the relationships they can represent grows as well. This is especially true for complex problem-solving. Logical problems, often represented as code or mathematics, can require models to combine and use a huge number of different patterns and a great deal of background information. Larger models have more capacity to represent these patterns than smaller ones, so increasing the size of a model is one important way of enhancing its capabilities [5].
A Recipe for an LLM
If you wanted to train an LLM of your own, there are three things that you would need:
- Plenty of model parameters, in the right shape (sometimes called the model's architecture)
- Lots of powerful computers (the amount of processing power going into a model is sometimes just called compute)
- Lots of training data (sometimes called a training corpus)
Understanding each of these ingredients will help to explain why the industry looks and acts the way it does.
Doing Lots of Maths Quickly — Graphics Processing Units
We've already talked about the number of parameters in these models, so we'll continue on to compute. For the sake of brevity, this series has not discussed the actual maths behind machine learning. For most people, the specific calculations are not all that important, but to really make sense of the AI boom, there are some things about them that you do need to know.
In truth, if you really wanted to and had more than a human lifetime, you could do all the maths to predict the next token in a sequence with pen and paper. Mathematically, no single operation in an LLM is really all that complicated. The problem is that with just one brain and one writing hand, you would have to do every single calculation one after another, in sequence. We call this a serial operation.
Most operations in an LLM don't actually require this, though. Consider the following equation: 3 × 2 + 1 = ?. To solve this equation, you have to perform each operation one at a time. First, 3 × 2 = 6. Then, 6 + 1 = 7. Each operation depends on the last one.
On the other hand, consider 2 × 3 + 4 × 5 = ?. In this case, there are three calculations to be made:
- 2 × 3 = 6
- 4 × 5 = 20
- 6 + 20 = 26
Notice that you don't need the answer for Step 1 in order to get the answer for Step 2. If you had a friend doing the maths with you, you could do Step 1 while they did Step 2, and once you were both done, you could move to Step 3. Steps 1 and 2 can be done in parallel, which saves you the time of doing one and then the other.
In machine learning, we very much want to make everything as parallel as possible. The Graphics Processing Unit (GPU) is a special component in a computer designed to do maths in parallel [6]. No single worker in a GPU is terribly complex, but there are thousands of them, all able to calculate at the same time. This is important because prior to the use of GPUs, machine learning was a prohibitively slow process. Detecting a single bird with our little five-million-parameter model might take something like 200 million individual calculations, which is just too many to do one at a time. GPUs, performing many operations in parallel, make the process of both training and using ML models substantially faster. For multi-billion-parameter language models, you could be looking at billions of individual calculations to generate just one token. This is why, if you hear people talking about the computers running AI models, the GPUs in those computers are the most important factor.
Training Data
The third ingredient you'll need for your LLM is training data, and a lot of it. Generally speaking, the more parameters you have, the more training data you need. Recall from our bird classifier that the training data we needed was a stack of photos with and without birds, and labels indicating which was which. The photos are the input, the labels are the output we expect. In the case of a model answering "which token comes next?", our input will be all the tokens so far, and our output will be the very next one.
Helpfully, this means that we can take a single sentence and split it into as many input/output pairs as there are tokens in the sentence.
Large models, though, mean lots of training data, so where do we get it? There are a few criteria we care about:
- We want an absolutely massive quantity of text.
- We want it to reflect patterns in real language, meaning we'd prefer a good deal of it to be written by real people.
- We want it to contain as much information about the world as possible, from a variety of different perspectives, so that our model can predict words about as many topics as possible.
Bigger models tend to be better at making predictions, but they also need a lot more data to train on, so at this point, most people training LLMs will take very large amounts of text from a wide variety of places. The best sources will typically be books (especially textbooks) and academic papers because, compared with other media, they are usually fairly well-written and have hopefully been checked by plenty of people to make sure what they say is true.
This is "high quality" data, but unfortunately, writing books takes a lot of time and effort, so there aren't nearly as many tokens in textbooks as there are on the internet. We'll use quite a bit of the public internet too. A corpus might include Wikipedia articles, forum posts, personal websites, and comment sections. If we want LLMs to write programs, then we'll add a large amount of code as well [7]. There is quite a bit of filtering involved here too, though. Labs training LLMs will usually try to remove duplicate material (we don't need one thousand identical copies of Hamlet; just one is enough), filter out garbage or unsavoury content, and so on.
I recognise that there is an elephant in the room with "What about copyright?" painted on its side. This series is intended to cover the technical side of LLMs, so for the moment we will not be addressing this problem. Exactly how we acquire training data (or the compute, for that matter) is a legal, political, and economic question for another day.
Pre-Training
So we have our model: the number of parameters and the architecture we want. We have our compute: the GPUs capable of performing highly parallel calculations. And we have our training data: a huge set of many trillions of tokens, written by people as much as possible. From these, we will create a usable LLM in two stages: pre-training and post-training.
Pre-training is very straightforward. We take our model and train it to predict the next token, using all the text we've accumulated. This can take days, weeks, or even months, depending on the size of the model and the training data, the amount of compute we have, the exact training settings we use, and whether we need to tweak those settings and try again. Pre-training is typically when the model sees the bulk of the actual data it will ever train on. It contains vast quantities of information, and so most of what a model "knows" is trained into it during this stage.
Post-Training
Once we're satisfied that our LLM trained on a large, processed mixture of internet data and other sources is sufficiently not-wrong about its predictions, we have what is sometimes called a "base" model. The problem is that virtually none of that text will look much like a conversation with ChatGPT or Claude [8]. We want our LLM to produce text that looks something like this:
PERSON: "You are a helpful AI assistant. What is the capital of France?"
ASSISTANT: "The capital of France is Paris. Would you like me to help you plan a visit?"
Most people will not have written anything like that by hand, so there will be very little for our base model to have learned. Without teaching it how to talk like an AI assistant, it might end up saying something like:
PERSON: "You are a helpful AI assistant. What is the capital of France?"
ASSISTANT: "The capital of France is:
1. Berlin
2. Paris
3. London
4. Rome"
In a case like this, the training data might have included a large amount of text from quizzes or homework, so the model continued the prompt as though it were writing a quiz rather than answering as an AI assistant. To solve this problem, we need to teach our model how we want an AI assistant to actually talk. This is the post-training phase. If the pre-training phase taught the model what it knows, the post-training phase will teach it how we want it to behave.
Post-training can involve several approaches, often used in combination:
- Write a large corpus of assistant-style interactions manually and train the base model on them. This would involve a huge number of manually written conversations, so it may supplement the main approach rather than replace it.
- Get the model to produce output, have a large team of people grade how closely it resembles the output of a good assistant, and train the model to be as not-wrong as it can about how we want it to sound.
- Get the model to solve mathematical problems, check the answers automatically with a calculator, and train it to be not-wrong about the answers.
- Get the model to produce code, check it automatically with code tests, and train it to be not-wrong about code that passes those tests.
Companies training LLMs aim to encourage many different types of behaviour. I intend to discuss them in detail in a later article , but some might include:
- Talking to users in a warm and friendly manner
- Telling the truth about facts it has good information about
- Refusing requests for illegal material and avoiding offence
- Resisting attempts by users to bypass those restrictions
- Using the particular textual structure we want when asking to use external tools
The way this feedback is provided will differ depending on the task, but it is ultimately used to update the model's weights, making assistant-like or correct responses more likely while undesirable ones become less likely.
Failure and Success
While making models gigantic and subjecting them to extensive post-training has resulted in some remarkable systems, at the end of the day, LLMs are still trained to predict likely continuations to their input text. They do not possess some readable, internal list of facts and rules that either they or we could refer to; all they have are patterns and predictions. Just because a token is likely to come next does not mean that the sentence produced by selecting that token is actually true or helpful. This is one of the most common failure modes that LLMs of any size suffer, and we will discuss it next time.
[1] AlexNet, which was a historically significant image recognition model published in 2012, had about 60,000,000 parameters. This was very large for the time.
[2] Many LLMs this big are not designed to use all of their hundreds of billions of parameters at the same time. For example, GLM 5.2, represented in the size comparison, only activates 40 billion parameters per token, though this is still an enormous number.
[3] "Small Language Model" doesn't really have a hard-and-fast definition. Ten billion parameters is my estimate for when people would probably stop calling them small, but it's a relative term, so that number may grow if top LLMs get much bigger, or shrink if they get smaller.
[4] See Smolka, Zeil, and Hemmi, Natural visual cues eliciting predator avoidance in fiddler crabs.
[5] This does get a little complicated. LLMs are not like hard drives in that they have measurable "used" and "free" space. The information they contain will be spread throughout their weights, and often in ways that are hard to explain or keep track of in human terms. Nonetheless, it remains true that a greater number of parameters means a greater potential for representing patterns.
[6] GPUs contrast with CPUs (Central Processing Units). CPUs are designed to do complex tasks and can usually do them fairly quickly, but do not have as many workers able to act at the same time as GPUs. The "Graphics" in GPU comes from the fact that many tasks requiring drawing things to the screen can also be done in parallel, particularly gaming and digital art or sculpting work.
[7] Interestingly, learning to write code also has the upside of teaching LLMs something of logical structure, which can help them in other domains as well. See Petty et al. and Aryabumi et al.
[8] At least, when they were first trained, it didn't. Nowadays there is a huge amount of LLM-generated text on the internet. Generally speaking, we prefer to keep raw chatbot logs out of our pre-training corpus for privacy and quality reasons. Cleaning this out is also a major challenge. These days, trainers do intentionally include synthetic, LLM-generated text in both pre-training and post-training, but they try to ensure that it is high-quality and unique.