Fundamentals
Token
The small chunks of text — often pieces of words — that a language model actually reads and generates, one at a time.
Models don't process raw letters or whole words; they break text into tokens — typically about 3-4 characters of English on average — using a fixed vocabulary learned during training. “Tokenization” is the process of converting text into these units, and back into text again when the model responds.
Tokens matter commercially, too: nearly every AI API prices usage per token, and a model’s context window is measured in tokens rather than words or characters — which is why “how many tokens” is the first question in estimating what an AI feature will cost to run.