A complete makeup look feels like one thing when you first see it.
But if you analyze it, you can separate foundation, brows, eyeshadow, lashes, blush, lip color and tiny details.
A Large Language Model (LLM) handles text in a similar spirit. It does not always treat a whole sentence—or even a whole word—as its smallest unit.
Instead, text is split into smaller pieces called tokens.
A token is not always one word or one character
This is the part that surprises most beginners.
A common English word may be one token. A longer or unusual word may be split into several. Chinese characters, punctuation and numbers can also be tokenized differently depending on the model.
So “500 characters” and “500 tokens” do not mean the same thing.
A full makeup look may look like one style, but once you break it down there can be twenty separate steps.
Why split text at all?
A model needs a consistent way to turn language into units it can process. The rules and software that perform this splitting are often called a tokenizer.
If every complete sentence were treated as one unique unit, the number of possible sentences would be practically endless. Smaller reusable pieces are much more manageable—like wardrobe basics that can appear in thousands of different outfits.
Why should you care about tokens?
You do not need to count them every day, but the idea matters for three reasons.
First, a model cannot process unlimited text at once. Long documents and long conversations use up tokens.
Second, many AI APIs charge based on the number of input and output tokens.
Third, tokens remind you that a language model does not process text exactly the way a person experiences a complete sentence.
Does ChatGPT generate one token at a time?
At a basic level, yes: an LLM predicts what token is likely to come next based on the context already present, then continues from there.
That is why the text can appear progressively on screen. The generation itself is sequential.
This leads naturally to the next idea: the prompt, because what you provide before the answer changes what comes next.
One thing to remember today
A token is a small unit used by a language model to process text. A sentence is broken into tokens before the model works with it.