Review Token

A token is one of the small pieces an input is broken into before an AI model can process it. In text, a piece may be a whole word, part of a word, a punctuation mark, or a space — the model reads and produces these pieces rather than whole words. The program that does the breaking up is called a tokenizer.

Before a model can work with your text, the text is chopped into small pieces. A common word like "the" is usually one piece; an unusual name or a long word can be three or four. Everything is counted in these pieces: how much you can fit into one conversation, how long an answer can run, and — on services that bill by usage — what you pay.

Token counts affect how much material fits in a context window, how long a response can be, and how some services calculate usage.

A long document consumes thousands of tokens when placed into a conversation, leaving fewer tokens available for instructions and the model's answer.

A token is not reliably equal to one word. The relationship changes with language, spelling, punctuation, content type, and tokenizer.

Tokens are the pieces the model processes, not a fixed word count.

* [[context-window|Context Window]] * [[large-language-model|Large Language Model]] * [[input|Input]] * [[output|Output]]

* [Google for Developers, Machine Learning Glossary](https://developers.google.com/machine-learning/glossary) — technical definition of tokens and tokenization in machine learning.

* See how tokens accumulate inside a [[context-window|Context Window]].