Ranking and calibration

RankerCopy Anchor Link

The ranker calculates two scores for each model in the reference bank: the frequency ranking $f$ and the position/order discriminant $q$. It standardizes each score across the candidates and adds them with equal weights:

$$s = 0.5\, z(f) + 0.5\, z(q), \qquad f = 0.5\, z_{\text{LDA}} + 0.25\, z_{\text{kNN}} + 0.25\, z_{\text{centroid}}$$

The frequency ranking $f$ has three parts:

• LDA: a linear discriminant projection of the features of the first 128 integers, averaged over the answers.
• Nearest neighbors: the mean of the 7 smallest squared distances to the reference answers of the model, with the median over the answers.
• Nuisance-free centroids: the prompt environment shifts the whole distribution of a model. When the bank is built, SVD estimates a nuisance subspace of at most 2 dimensions from the mean offsets of the environments. At scoring time, the ranker first projects out this subspace, then calculates the cosine similarity to each model centroid. The position-block features are also compared with the model templates of each environment.

The position/order discriminant $q$ is an LDA with shrinkage 0.5. It uses four feature blocks with 751 dimensions in total:

Each block is standardized and normalized to unit length, then concatenated with its weight. The ranker standardizes the discriminant values of each answer across the candidates, then averages them over the answers.

The ranking score $s$ sets the candidate order. The weights of the two scores and the shrinkage come from leave-one-environment-out cross-validation on the reference bank. The fixed evaluation set does not take part in this selection.

Confidence calibrationCopy Anchor Link

The confidence is calculated only when all three answers are valid and the calibration in data/shared_detector.bin matches the detector. It is a temperature softmax within the reference bank:

$$p_k = \frac{\exp(\tau s_k)}{\sum_j \exp(\tau s_j)}$$

fpd retrain fits the temperature $\tau$ offline. It holds out each of the 12 environments, then each pair of environments, and refits the ranker 78 times. The held-out results give the estimate of $\tau$. The nested held-out binary NLL must be lower than a constant baseline, and the AUC must be greater than 0.75. Otherwise, retraining does not replace the current detector. $p_k$ is distributed only among the models in the reference bank: all candidates add up to 100%, and models outside the bank are not included. It is not the probability that the service really is that model. When the calibration does not match the detector, the result gives the ranking only, without confidence.

Reference bank and detectorCopy Anchor Link

If the reference bank does not match the detector (for example, a custom bank from --bank), detection stops with an error; run fpd retrain --data-dir DIR first. See Data files for the format and change log of the data files.