Tokens are the atomic units of text that language models read, process, and generate. They are not characters, not words, and not sentences — they are subword fragments determined by a tokenization algorithm that balances vocabulary size against encoding efficiency. The word “understanding” might be a single token, while “misunderstanding” might be split into [“mis”, “understanding”]. Common words like “the” and “is” are single tokens; rare words and technical terms get split into smaller pieces.
Tokens are the universal currency of the AI industry. They determine what a model can read (context window size is measured in tokens), what it costs to use (API pricing is per-token), how fast it responds (generation speed is measured in tokens per second), and what it can reason about (the model literally cannot see below the token level). Understanding tokenization is essential for anyone building, using, or evaluating language model applications.
How tokenization works
Modern language models use subword tokenization algorithms that sit between character-level and word-level encoding. The dominant algorithm is Byte Pair Encoding (BPE), with SentencePiece as a common alternative that operates on raw bytes rather than Unicode characters.
Byte Pair Encoding (BPE)
BPE builds a vocabulary through an iterative merging process:
- Start with a base vocabulary of individual bytes (256 entries) or Unicode characters
- Count all adjacent pairs of tokens in the training corpus
- Merge the most frequent pair into a single new token
- Repeat steps 2-3 until the vocabulary reaches a target size (typically 32K-200K tokens)
The result is a vocabulary where common words and subwords are single tokens, while rare words are decomposed into smaller pieces. The algorithm is deterministic — the same text always produces the same token sequence for a given tokenizer.
Tokenizer specifications across providers
Different model families use different tokenizers, which means the same text produces different token counts across providers:
| Tokenizer | Used by | Vocabulary size | Avg chars/token (English) | Notable features |
|---|---|---|---|---|
| cl100k_base | GPT-4, GPT-4 Turbo | 100,256 | ~4.0 | Improved multilingual over previous versions |
| o200k_base | GPT-4o, GPT-4.1, o1, o3 | 200,019 | ~4.2 | 2x vocabulary; better multilingual efficiency |
| SentencePiece (custom) | Claude family | ~150K (est.) | ~3.8 | Byte-level fallback for any Unicode |
| SentencePiece | Gemini family | ~256K | ~4.3 | Large vocabulary for multilingual coverage |
| SentencePiece | Llama 3/4 | 128,256 | ~4.1 | Extended from Llama 2’s 32K vocabulary |
| SentencePiece | Mistral | 32,768 | ~3.5 | Smaller vocabulary, optimized for European languages |
The practical implication: a 1,000-word English document might tokenize to 1,300 tokens on one model and 1,450 on another. This 10-15% variance matters for cost estimation and context window planning.
What tokens look like in practice
Understanding tokenization at a concrete level reveals its quirks and consequences:
English text (efficient): “The cat sat on the mat” = 6 tokens: [“The”, ” cat”, ” sat”, ” on”, ” the”, ” mat”]. Note: leading spaces are typically included in the token.
Technical text (less efficient): “PostgreSQL” = 3 tokens: [“Post”, “gre”, “SQL”]. Technical terms and proper nouns that are rare in training data get split into smaller pieces.
Code (variable): Python code tokenizes reasonably efficiently because code is well-represented in training data. Indentation whitespace is typically encoded as a single token per indent level. Special characters and unusual syntax patterns use more tokens.
Non-English text (often inefficient): Chinese, Japanese, Korean, Arabic, and other non-Latin scripts are typically tokenized less efficiently. A Chinese sentence equivalent to 10 English words might require 2-3x more tokens, because each Chinese character may become one or more tokens rather than being grouped into multi-character tokens.
Numbers: Numeric sequences are tokenized inconsistently. “1000” might be a single token, but “10000” might be [“100”, “00”]. This is one reason language models struggle with arithmetic — they process numbers as subword sequences, not as numerical values.
The token economy: pricing and costs
Tokens are the billing unit for all major language model APIs. Pricing is quoted per million tokens, with input (prompt) tokens typically cheaper than output (completion) tokens because output requires autoregressive generation while input can be processed in parallel.
Frontier model pricing (as of mid-2026)
| Model | Input (per 1M tokens) | Output (per 1M tokens) | Effective cost per word (output) |
|---|---|---|---|
| Claude Opus 4 | $15.00 | $75.00 | ~$0.0001 |
| Claude Sonnet 4 | $3.00 | $15.00 | ~$0.00002 |
| Claude Haiku 3.5 | $0.80 | $4.00 | ~$0.000005 |
| GPT-4.1 | $2.00 | $8.00 | ~$0.00001 |
| GPT-4.1 mini | $0.40 | $1.60 | ~$0.000002 |
| GPT-4.1 nano | $0.10 | $0.40 | ~$0.0000005 |
| Gemini 2.5 Pro | $1.25-$2.50 | $10.00-$15.00 | ~$0.00001 |
| Gemini 2.5 Flash | $0.15 | $0.60 | ~$0.0000008 |
| Llama 4 Maverick (self-hosted) | Compute cost only | Compute cost only | Varies by hardware |
The 100-200x price difference between frontier models (Opus 4) and efficient models (GPT-4.1 nano, Gemini Flash) drives architectural decisions in production applications. Many systems use model cascading — routing simple queries to cheap models and escalating complex queries to expensive ones — to optimize the cost-quality tradeoff.
Prompt caching: reducing input token costs
Both Anthropic and OpenAI offer prompt caching, which stores the computed representation of repeated prompt prefixes. When a cached prefix is reused, input token costs drop by 75-90%:
| Provider | Caching feature | Cache discount | Cache TTL |
|---|---|---|---|
| Anthropic | Prompt caching | 90% discount on cached tokens | 5 minutes (auto-extended on hit) |
| OpenAI | Automatic caching | 50% discount on cached tokens | Up to 1 hour |
| Context caching | 75% discount on cached tokens | Configurable |
For applications with long system prompts (1,000-10,000 tokens) that remain constant across queries, caching transforms the economics. A 5,000-token system prompt on Claude Sonnet 4 costs $0.015 per query without caching but only $0.0015 with caching — a 10x reduction on the prompt portion.
Tokens and model capabilities
Token mechanics directly shape what language models can and cannot do:
Context windows
A model’s context window is the maximum number of tokens it can process in a single request (input + output combined). Context window sizes have expanded dramatically:
| Year | Frontier context window | Example model |
|---|---|---|
| 2022 | 4,096 tokens (~3,000 words) | GPT-3.5 |
| 2023 | 32,768 tokens (~25,000 words) | GPT-4 (32K) |
| 2023 | 100,000 tokens (~75,000 words) | Claude 2.1 |
| 2024 | 200,000 tokens (~150,000 words) | Claude 3 Opus |
| 2025 | 1,000,000 tokens (~750,000 words) | Gemini 2.5 Pro |
| 2025 | 200,000 tokens | Claude Opus 4 |
Larger context windows enable new use cases — analyzing entire codebases, processing book-length documents, maintaining long conversation histories — but come with tradeoffs. Attention computation scales quadratically with sequence length (though Flash Attention and other optimizations reduce the practical impact), and models attend less precisely to information in the middle of very long contexts (the “lost in the middle” phenomenon).
Generation speed
Output generation speed is measured in tokens per second. A model generating 50 tokens/second produces roughly 37 words/second — noticeably faster than human reading speed. Key benchmarks:
| Model tier | Typical output speed | Time for a 500-word response |
|---|---|---|
| Frontier (Opus-class) | 30-60 tokens/sec | 10-20 seconds |
| Mid-tier (Sonnet/GPT-4.1) | 60-120 tokens/sec | 5-10 seconds |
| Fast (Haiku/Flash/mini) | 150-300 tokens/sec | 2-4 seconds |
| Speculative decoding enabled | 2-3x base speed | Proportionally faster |
For agentic applications that make dozens of model calls per task, output speed directly impacts task completion time. A coding agent that averages 30 model calls per task will spend 5-10 minutes on generation alone at Opus-class speeds, versus 1-2 minutes with Sonnet.
The character-blindness problem
Because models operate on tokens, not characters, they are fundamentally unable to “see” individual characters in most words. This creates systematic failure modes:
- Letter counting: “How many r’s in ‘strawberry’?” is famously difficult because the model sees [“str”, “aw”, “berry”] (approximately), not individual letters.
- String reversal: Reversing “hello” requires character-level manipulation that the model must simulate through reasoning.
- Anagrams and wordplay: Tasks requiring character permutation are unnatural for token-level processing.
- Spelling: The model can misspell rare words because it generates them as token sequences, not character sequences.
Models have improved at these tasks through training techniques — including training on character-level tasks and using chain-of-thought reasoning — but the underlying limitation remains architectural. Some applications work around this by splitting text into individual characters before sending it to the model.
The tokenization efficiency gap across languages
One of the most consequential aspects of tokenization is its uneven efficiency across languages. Tokenizers trained primarily on English text develop vocabularies optimized for English subword patterns. Other languages — especially those with different scripts, agglutinative morphology, or logographic writing systems — are tokenized less efficiently:
| Language | Tokens per equivalent meaning (relative to English) | Cost multiplier | Context window effective reduction |
|---|---|---|---|
| English | 1.0x (baseline) | 1.0x | None |
| Spanish, French | 1.1-1.3x | 1.1-1.3x | 10-25% |
| German | 1.2-1.4x | 1.2-1.4x | 15-30% |
| Chinese (Mandarin) | 1.5-2.5x | 1.5-2.5x | 35-60% |
| Japanese | 1.5-2.5x | 1.5-2.5x | 35-60% |
| Korean | 1.3-2.0x | 1.3-2.0x | 25-50% |
| Arabic | 1.5-2.5x | 1.5-2.5x | 35-60% |
| Hindi | 2.0-3.0x | 2.0-3.0x | 50-65% |
| Thai | 2.0-3.5x | 2.0-3.5x | 50-70% |
This means a Thai user pays 2-3x more than an English user for an equivalent conversation, and gets 50-70% less effective context window. Newer tokenizers with larger vocabularies (200K+) partially close this gap by allocating more vocabulary entries to non-English subwords. OpenAI’s o200k_base tokenizer improved multilingual efficiency by 10-30% over cl100k_base for most languages.
Token counting in practice
Accurate token counting is essential for cost estimation, context window management, and rate limit planning. Tools and approaches:
Official tokenizer libraries:
- OpenAI:
tiktoken(Python) — fast, deterministic, supports all OpenAI tokenizers - Anthropic: API response headers include token counts; no public tokenizer library, but the API’s token counting endpoint provides exact counts
- Google:
sentencepiece(Python/C++) for Gemini-family models - Hugging Face:
tokenizerslibrary supports most open-source model tokenizers
Rules of thumb for estimation (English):
- 1 token is approximately 4 characters or 0.75 words
- 1 page of text (single-spaced) is approximately 500-700 tokens
- 1,000 words is approximately 1,300-1,500 tokens
- A 300-page book is approximately 100,000-150,000 tokens
These approximations are accurate to within 15% for standard English prose. Technical text, code, and non-English text deviate more significantly.
Special tokens and control sequences
Beyond content tokens, models use special tokens for structural purposes:
| Special token | Purpose | Example |
|---|---|---|
| BOS (beginning of sequence) | Marks the start of input | <s> or <|begin_of_text|> |
| EOS (end of sequence) | Signals the model to stop generating | </s> or <|end_of_text|> |
| PAD (padding) | Fills unused positions in batched inputs | <pad> |
| System/User/Assistant markers | Delineate conversation roles | <|user|>, <|assistant|> |
| Tool use markers | Indicate tool calls and results | Model-specific formats |
These tokens are typically invisible to end users but matter for developers building with model APIs. They consume part of the context window — conversation formatting overhead (role markers, delimiters) typically adds 50-200 tokens per turn, which can accumulate significantly in long conversations.
Frequently asked questions
How are tokens different from words? Words are linguistic units separated by spaces; tokens are subword units determined by a mathematical algorithm (typically BPE). Common words like “the” are single tokens, but longer or rarer words are split into multiple tokens. The word “tokenization” might be two tokens: [“token”, “ization”]. On average, one English token equals about 0.75 words, but this ratio varies by text type and language.
Why do different AI models have different token counts for the same text? Each model family uses a different tokenizer with a different vocabulary. OpenAI’s GPT-4.1 uses o200k_base (200K vocabulary), Llama 4 uses a 128K SentencePiece vocabulary, and Claude uses its own tokenizer. Different vocabularies split text differently, producing different token counts. The variance is typically 10-15% for English text and can be larger for other languages.
How can I reduce my token costs? Five approaches, in order of impact: (1) Use prompt caching for repeated system prompts (saves 50-90% on input tokens). (2) Use the smallest model that meets your quality bar — Haiku/Flash/mini models are 10-50x cheaper than frontier models. (3) Compress prompts by removing redundant instructions and using concise language. (4) Implement model cascading to route simple queries to cheap models. (5) Use batch APIs (available from most providers at 50% discount) for non-real-time workloads.
Why are output tokens more expensive than input tokens? Input tokens are processed in parallel — the model reads the entire prompt in one forward pass. Output tokens are generated one at a time (autoregressively), with each token requiring a separate forward pass through the model. This sequential generation is computationally more expensive per token than parallel input processing, which is reflected in the pricing differential (typically 3-5x for most providers).
What happens when I exceed the context window? If your input exceeds the model’s context window, the API returns an error. In chat applications, older messages are typically truncated or summarized to stay within limits. Some frameworks implement automatic context management — sliding windows that drop the oldest messages, or summarization of conversation history. For applications processing long documents, techniques like chunking (splitting the document and processing pieces separately) or using models with larger context windows (Gemini 2.5 Pro at 1M tokens) are necessary.