Token

Language models don't process text character by character or word by word — they use a tokenizer to split text into tokens. "hamburger" might be one token, while "antidisestablishmentarianism" might be five. Spaces, punctuation, and special characters are often their own tokens.

Understanding tokens is critical for: (1) Estimating API costs — Claude Sonnet costs $3/M input tokens + $15/M output tokens. (2) Managing context windows — 200K tokens ≈ 150,000 words ≈ 500 pages of text. (3) Debugging truncation issues when prompts get cut off.

Rule of thumb: 1,000 tokens ≈ 750 words. A typical business email is ~300 tokens; a 10-page report is ~4,000 tokens; an entire book is ~75,000 tokens.