Token Calculator

Example#

The tool opens with a short sample prompt and counts its tokens with the o200k_base encoding. The result shows the token count marked Exact, plus characters, words and lines. If you pick a model that has a known context window, it also shows how much of that window the text uses.

How it works#

For OpenAI's published encodings, o200k_base and cl100k_base, the tool runs a JavaScript port of the tokenizer in a Web Worker, so typing stays smooth with long text. The encoding data loads only when you first need it. The result is labeled Exact because it is the real token count for that encoding. Special-token strings such as <|endoftext|> are counted as ordinary text.

For any other model, the tool shows a labeled estimate: the number of characters divided by four, rounded up, which is OpenAI's published rule of thumb for English text. Characters are counted as Unicode code points, words are runs separated by whitespace, and lines are line breaks plus one.

Assumptions#

• Exact counts are only available for the two OpenAI encodings.
• The text is at most 500,000 characters; longer text is not processed.
• When a context window and an input price are known for the chosen model, the tool also shows the share of the window used and the input cost of this text.

Use cases#

• Checking that a prompt fits in a model's context window.
• Estimating the input cost of a long document.
• Comparing how the same text tokenizes with two encodings.
• Trimming a prompt to a token budget.

Limitations#

A token count is exact only for the encoding you choose. Chat formatting, system prompts, images and tool definitions add tokens that this count does not include. Estimates for other providers can be off by a wide margin, especially for code and for languages other than English. Your text is processed in your browser and is not sent anywhere.