Tokenizer probe
While sampling, API detection identifies the upstream tokenizer and checks whether it matches the model you entered. The result is reference information; it neither changes the ranking nor delays the detection result.
A tokenizer is the rule a model uses to split text into tokens. Models from one vendor usually share a tokenizer, so knowing which one the upstream (the service behind the address you entered) uses is a useful cross-check on the number fingerprint's ranking. It also exposes a channel that claims gpt-5 while serving a Qwen tokenizer.
The tokenizer probe is reference information only: it appears on its own in the "Tokenizer probe" card, the fourth entry of the sample strip, and the tokenizer details. It does not change the candidate ranking or the confidence, and the detection result does not wait for it.
The tokenizer probe is part of API mode and is off by default. Turn on the "Tokenizer probe" switch in the API configuration to probe while sampling. It costs about 12 extra short requests while sampling, at most 22; APIs billed per request bill these requests. They reuse your service tier and your reasoning-effort setting (effort except on Messages) and, apart from Messages, set no output limit, so a reasoning model can spend noticeably more tokens. Manual mode has no API to ask, so it does not probe.
What happens during a detection
Start sampling
After you click "Start sampling", a "Tokenizer probe" card appears below the three sample cards. It shows the number of finished requests and the tokenizer leading so far. Each request only asks the model to reply OK; the page reads the input token count from the usage the API returns.
With "Parallel requests" on, the tokenizer probe sends four requests at a time; with it off, requests go one by one, which takes longer.
The three samples share one probe: stopping a sample card ends only that sample, and the probe keeps running; "Stop" in the action bar also stops the probe.
Read the result
When the three samples finish, verification computes and shows the result at once, without waiting for the probe. A probe that is still running continues beside the result: the fourth entry of the sample strip, "Tokenizer", shows "Probing", updates as new counts arrive, and opens the details. "Stop probe" in the action bar ends the probe at any time, and the page gives its verdict from the counts received so far.
Tokenizer details
The top of the details holds the title, "Collapse", the status badge and the More actions menu. One line below it says that the result is reference only. If the probe fails, is stopped, or the two baseline requests count differently, a notice appears under that line.
The body has two columns. On narrow screens, the blocks stack in this order:
| Position | Content |
|---|---|
| Top left | The verdict: "Upstream tokenizer" ("Leading so far" while probing), the series name, the lab name and the confidence |
| Top right | "Model you entered": the model ID and the check result, see Check against the model you entered |
| Bottom left | Candidate tokenizers: the confidence of the top 6 classes; the last row is always "Unlisted tokenizer" |
| Bottom right | Request log: each row is a probe text or the "Baseline request" (only the fixed prefix and suffix), with its input token count and the tokens added over the baseline |
There are three verdicts, shown on the status badge:
| Verdict | Meaning |
|---|---|
| Exact match | The counts of nearly all probe texts agree with one listed tokenizer class, allowing a few deviations |
| Close to a listed class | Close to one class, but a sizable share of probe texts count differently. It may be a derivative or an extension of that tokenizer |
| Not listed | Unrelated to every listed class |
A class groups open tokenizers that count almost identically. This tool lists 62 classes; see Listed tokenizers.
The bottom row shows the request count and the duration, with "Probe again" on the right. Below it are three collapsible sections: the open tokenizers in this class, the counts per probe text, and "How to read the result".
Check against the model you entered
Once the probe ends, the details compare the model name you entered with the vendor model mapping:
| Result | Meaning |
|---|---|
| Match | The measured tokenizer is the one the model uses |
| No match | For example, the channel claims gpt-5, but the probe measures the Qwen tokenizer |
| Counts cannot decide | The counts have not yet concentrated on one class |
| Tool does not know | This tool does not know which tokenizer the model uses |
Vendors such as Anthropic (Claude) and xAI (Grok) have not published their tokenizers: an unlisted tokenizer counts as consistent, and any listed class counts as inconsistent. Vendors such as OpenAI and Google have published theirs, so their models are consistent only with an exact match; "Close to a listed class" does not count. For open models matched by an alias, an unlisted tokenizer close to their class also counts as consistent.
More than one model can share a tokenizer
One tokenizer is often reused by several models. For example, Xiaomi MiMo and MiniCPM-V reuse the Qwen1 to Qwen3 tokenizer. Some relays estimate usage locally with tiktoken, OpenAI's open-source tokenizer library, which also produces o200k_base or cl100k_base counts; when the verdict is one of these and does not match the model you entered, the "Model you entered" column mentions this possibility. When the two baseline requests count differently, the page warns that the upstream adds hidden input of varying length, such as a system prompt; the result may be inaccurate, so probe again to check.
When there is no tokenizer result
- The API response has no usage. Some relays do not pass the upstream usage on; this is a limit of the API itself.
- The baseline request fails, or no probe text is answered. When probing fails or is stopped partway, the verdict uses the counts received so far.
When probing fails or is stopped, use the More actions menu of the "Tokenizer probe" card or of the details, or "Probe again" at the bottom of the details. It reuses the configuration of the last probe, does not sample again, and leaves the detection result on screen unchanged.
Export
"Export JSON" in the More actions menu of the result contains the candidate list from the number fingerprint alone. Once the probe has ended, it also carries the probe report under tokenizer: the verdict, the check against the model you entered, and the count of every request. The file contains neither the API address nor the API key.
Last updated on
Website
The manual and API sampling modes, the switches in the API configuration, and how to read detection results.
Connection
The Connection setting decides whether the browser calls the API directly or sends requests through this site's proxy or a Worker you deploy yourself. By default, requests go through this site's proxy. Auto first checks whether the API allows web pages to call it directly and otherwise uses this site's proxy.