Tokenizer probe

# Detect and run the tokenizer probe (off by default)
npx lmfpd@latest -b https://api.example.com/v1 -k sk-xxx -m deepseek-v4-pro -a cc --tokenizer

# Read the tokenizer probe saved in a report offline, without requests
npx lmfpd@latest --input result.json --json

A tokenizer is the rule that a model uses to cut text into tokens. Different model families use different tokenizers. Thus the input token count that an API reports in its usage (usage) can show the upstream tokenizer.

The tokenizer probe is off by default. With --tokenizer, fpd sends one baseline request and a set of probe texts beside the first round. It probes one time for each detection. A probe usually sends about 12 more requests and at most --max-probes + 2 (22 by default). APIs that bill per request bill these requests.

The probe result is reference only:

• The candidate ranking and the confidence come from the number fingerprint alone. They are the same with or without the probe.
• Each round is scored when its samples finish. Scoring does not wait for the probe.
• A matching tokenizer does not prove the model identity, because many models share one tokenizer.

If --parallel is above 1, the probe sends four requests at a time. Otherwise, it sends one request at a time. Each request has a 90-second deadline. Messages requests limit the output to 16 tokens. Like sample requests, the other protocols set no output limit. For the website, see Tokenizer probe on the website. For the method, see How it works.

OptionsCopy Anchor Link

If you use --max-probes or --tokenizer-bank without --tokenizer, fpd stops with an error before it sends a request. --input sends no requests, so you cannot use it with --tokenizer or --max-probes.

The connection options are the same as for the detection command. fpd checks the tokenizer against the model name from -m. For example, if the model name is gpt-5 and the probe finds the Qwen tokenizer, the interface shows no match, and claim.status in the report is inconsistent.

Terminal interfaceCopy Anchor Link

The TOKENIZER PROBE block is below the ranking. Its first line tells you that the block is reference only.

• During the probe, the block shows the number of answered requests and the tokenizer that leads on the counts so far.
• After the probe, the block shows the verdict, the lab of the tokenizer, and the check against --model.
• If the verdict is o200k_base or cl100k_base and the check result is not a match, the block tells you that the counts can come from an estimate by a relay.
• If the two baseline requests gave different counts, the block tells you that the result can be wrong.

If the probe is still running after the last round, the status line shows Ranking final · tokenizer probe running. The ranking is final at this point. fpd waits for the probe before it exits, so the interface and the report get the complete probe result. Press q or Ctrl+C to stop the probe and exit:

• fpd keeps the results of all rounds, and --output saves the report.
• The tokenizer verdict uses the counts received before the stop.
• In the report, the top-level cancelled is false and tokenizer.cancelled is true.
• The exit code is 130.

If the output is redirected, fpd writes one line with the probe result to stderr.

Report formatCopy Anchor Link

With --tokenizer, reports from --output and --json contain a tokenizer object:

The tokenizer field in the request settings of the report records whether --tokenizer was set. The analysis of each round comes from the number fingerprint alone. The report contains no API key.

Offline readingCopy Anchor Link

--input reads tokenizer.observations from a report and recomputes the result with the current tokenizer bank. It sends no requests and does not need --tokenizer. Use --tokenizer-bank to select a different tokenizer bank. Use -m or MODEL to replace the model name saved in the report.

If the current tokenizer bank does not contain a probe text, fpd skips its count and shows one warning. With --json, the warning goes to stderr. If no count remains after the skip, the interface shows no tokenizer result, and the exit code does not change. A report saved again with --output keeps all original counts.

Failures and exit codesCopy Anchor Link

If the probe fails partway, the verdict uses the counts received before the failure. In these cases, there is no tokenizer result, and detection completes:

• The API response has no usage.
• The baseline request fails.
• No probe text gets an answer.

A failed probe does not change the exit code.