Model Lifecycle
Training (Pretraining)
The process of teaching a model its base knowledge and language ability by having it predict text across a huge dataset.
During pretraining, a model repeatedly guesses the next token in text pulled from books, websites, code, and more, adjusting billions of internal parameters (weights) each time it's wrong. This is the most compute-intensive phase of building an LLM, run across thousands of GPUs or TPUs for weeks or months at a time.
Pretraining gives a model general capability, but it doesn't yet make the model good at following instructions or safe to talk to — that comes from fine-tuning afterward.