Training a model on a large dataset before fine-tuning it to a specific task. Example: Pre-training a language model on a large corpus of text data before fine-tuning it for a specific natural language task such as language translation.
Pre-training is where almost all of a large language model’s capability comes from. Everything that happens afterwards — fine-tuning, instruction tuning, RLHF — shapes behaviour at the margins. Pre-training is what builds the thing being shaped.
How pre-training works
The defining feature is that it needs no labels. A model is shown enormous quantities of text and given a task it can grade itself on: predict the next token. Get it wrong, adjust the weights, repeat. Because the data supplies its own supervision, this is called self-supervised learning, and it is the reason pre-training can consume the open web rather than a hand-annotated dataset.
What emerges is not a model that has memorised the internet. It is a model that has been forced, billions of times, to compress patterns in language well enough to predict what comes next — and grammar, world knowledge, and some reasoning turn out to be useful compression.
Why pre-training matters
It is where the money goes. A frontier pre-training run costs tens to hundreds of millions of dollars in compute. This is the single biggest reason there are a handful of foundation model labs rather than hundreds.
It fixes the knowledge cutoff. A model knows what was in its pre-training corpus and nothing after. No amount of fine-tuning adds the news.
It sets the ceiling. Fine-tuning can surface and direct what a model already learned; it cannot install capabilities that were never there.
How you spend the budget matters too. DeepMind’s Chinchilla paper showed that models of its era were substantially under-trained for their size — that for a fixed compute budget, a smaller model trained on more data beats a larger model trained on less. That finding reshaped how labs size their runs.
Related terms: Fine-tuning, Foundation Model, Unsupervised Learning, Large Language Model (LLM), Tokenization.
