Skip to content
Get in touch
A page of writing enters a hopper machine. Seven separate colored tiles emerge, each holding a small text or punctuation fragment.
Reviewed visual · version 1

RS Glossary

Token

A token is one of the small pieces an input is broken into before an AI model can process it. In text, a piece may be a whole word, part of a word, a punctuation mark, or a space — the model reads and produces these pieces rather than whole words. The program that does the breaking up is called a tokenizer.

Review state
Reviewed in POC
Freshness
Current
Version
4
  • Token
  • Tokens

Remember it this wayTokens are the pieces the model processes, not a fixed word count.

In plain language

Before a model can work with your text, the text is chopped into small pieces. A common word like "the" is usually one piece; an unusual name or a long word can be three or four. Everything is counted in these pieces: how much you can fit into one conversation, how long an answer can run, and — on services that bill by usage — what you pay.

Why it matters

Token counts affect how much material fits in a context window, how long a response can be, and how some services calculate usage.

Example

A long document consumes thousands of tokens when placed into a conversation, leaving fewer tokens available for instructions and the model's answer.

Common misconception

A token is not reliably equal to one word. The relationship changes with language, spelling, punctuation, content type, and tokenizer.

Related terms

Sources

Learn more

  • See how tokens accumulate inside a [[context-window|Context Window]].

Public history

47 recorded events · 47 public summaries

  1. A potential shared concept was discussed and selected for initial glossary drafting.

  2. The captured candidate received a complete initial draft and entered section-level human review.

  3. Radar reviewed the initial definition draft for human consideration.

  4. Radar reviewed the initial plain-language-explanation draft for human consideration.

  5. Radar reviewed the initial why-it-matters draft for human consideration.

  6. Radar reviewed the initial example draft for human consideration.

  7. Radar reviewed the initial common-misconception draft for human consideration.

  8. Radar reviewed the initial remember-it-this-way draft for human consideration.

  9. Radar reviewed the initial related-terms draft for human consideration.

  10. Radar reviewed the initial sources draft for human consideration.

  11. Radar reviewed the initial learn-more draft for human consideration.

  12. A proposal was opened for section-level human review while preserving the current version.

  13. A human reviewer opened a proposed revision while preserving the current approved or captured version.

  14. definition approved by a human reviewer.

  15. plain-language-explanation approved by a human reviewer.

  16. why-it-matters approved by a human reviewer.

  17. example approved by a human reviewer.

  18. common-misconception approved by a human reviewer.

  19. remember-it-this-way approved by a human reviewer.

  20. related-terms approved by a human reviewer.

  21. sources returned for revision.

  22. learn-more approved by a human reviewer.

  23. Requested wording was prepared without changing approved or existing sections.

  24. Radar reviewed the pending sources section for human consideration.

  25. Radar must rereview the complete token term after an accepted text change.

  26. Radar must rereview the complete token term after an accepted text change.

  27. Radar must rereview the complete token term after an accepted text change.

  28. Radar must rereview the complete token term after an accepted text change.

  29. Radar must rereview the complete token term after an accepted text change.

  30. Radar must rereview the complete token term after an accepted text change.

  31. Radar must rereview the complete token term after an accepted text change.

  32. Radar must rereview the complete token term after an accepted text change.

  33. Radar must rereview the complete token term after an accepted text change.

  34. sources approved by a human reviewer.

  35. sources approved wording became the current section text.

  36. All section gates are approved; content version 2 is now a fully approved term.

  37. Radar rereviewed the human-approved definition wording for this exact version.

  38. Radar rereviewed the human-approved plain-language-explanation wording for this exact version.

  39. Radar rereviewed the human-approved why-it-matters wording for this exact version.

  40. Radar rereviewed the human-approved example wording for this exact version.

  41. Radar rereviewed the human-approved common-misconception wording for this exact version.

  42. Radar rereviewed the human-approved remember-it-this-way wording for this exact version.

  43. Radar rereviewed the human-approved related-terms wording for this exact version.

  44. Radar rereviewed the human-approved sources wording for this exact version.

  45. Radar rereviewed the human-approved learn-more wording for this exact version.

  46. Definition review recorded as replace.

  47. Plain-Language Explanation review recorded as replace.

Source: vault/Glossary/terms/token/term.md · sha256:7428e433ce2c1673151eb21260e9881fba73b770396e548bec18dd875c84ea92 · Release: rs-glossary-glossary-team-poc-0.5.5-2026-09-02