Tokenizer Lab
Before a model can reason about a sentence, it has to turn the sentence into integers. The tokenizer decides the units, the context budget and part of the bill. Here is the text the model actually receives.
o200k_base and cl100k_base tokenizers from gpt-tokenizer. Counts are exact for those vocabularies. Other model families use different tokenizers, so their counts will differ. Nothing you type leaves the browser.Loading the vocabulary and merging rules…
Why the split matters
A model does not see letters or words. It sees token IDs from a fixed vocabulary. The same sentence can cost different numbers of tokens under different vocabularies, and rare words, code identifiers, JSON punctuation and non-Latin scripts are often split into several tokens.
cost = input tokens × input price + output tokens × output price
Counts here are exact for the two OpenAI vocabularies. Llama, Qwen, Mistral and Gemini use different tokenizers, so their counts will differ. Tokenization is model-specific.
Find the expensive characters.
Tokenization is full of small surprises. Change one thing, then compare the token count and the split, not just the text.
A word is not a token.
Type “internationalization” and then “cat”. Both are single words. Compare their token counts and the visible pieces. Then paste a long identifier from your own code. Which one would you rather pay for a million times?
Code is punctuation.
Choose the JSON preset. Look at the braces, quotes and colons. They are structure to a parser and tokens to a model. Switch vocabularies and find where the split changes. This is part of why tool-call formats are designed to be easy to emit.
English is not neutral.
Choose Multilingual and compare characters per token. A sentence with the same meaning can cost very different token counts in different scripts. That is a product and fairness decision, not just a formatting detail.