Tiktokenizer: how a tokenizer splits text into tokens

What a tokenizer does

An LLM never reads a sentence whole. Text first passes through a tokenizer, which splits it into token fragments and maps each fragment to an integer IDfrom the vocabulary. What the model actually manipulates is those numbers, not words. Type a sentence into the tool above to watch where each model’s tokenizer draws its boundaries and how token counts and IDs change in real time.

Why try it yourself

• Cost: LLM APIs bill per token. The same content at a different token count is a different bill.
• Context limits: context windows are measured in tokens, not characters. Managing the limit starts with a feel for how many tokens your prompt takes.
• Language differences: Korean tends to spend more tokens than English for the same meaning. Try the same sentence in both languages and compare.
• Model differences: tokenizers differ per model, so the same sentence yields different token counts. Switch models in the dropdown and watch the boundaries move.

Where to go next

The other two entries in this series cover what happens as token counts grow: part 1 visualizes how attention comparisons explode at N² and disperse the model’s focus, and part 2 explains Lost in the Middle, where information in the middle of the context gets dropped.