0
$\begingroup$

I am comparing 5 third party classification models on a subset of results (specifically, false positives I am examining to find a common cause). The five models all output values between 0 and 1 but seem to have different overall sensitivities.

I could try and calculate a skew value for each model's distribution from the entire dataset to normalise it, or I could normalise the classification only within the range of outputs associated with the subset values. (Note that this is a heavily uneven dataset.)

Improve this question
$\endgroup$
2
  • $\begingroup$ Hi, Jess! If the outputs are between 0 and 1, then they are not making classifications. How do you get classifications from those raw outputs? There is a default method (despite its flaws), there are alternative methods (which also have flaws), and then there is a school of thought that the raw outputs should be assessed without making any classifications. $\endgroup$
    – Dave
    Commented Oct 26 at 15:00
  • $\begingroup$ @Dave thanks for your reply, so the approach I used so far was to calculate the optimum threshold by calculating F1 along the precision-recall curve to turn a "probability" output into a class $\endgroup$
    – Jess
    Commented Oct 28 at 8:51

1 Answer 1

Reset to default
1
$\begingroup$

You should try doing Probability Calibration. Training a model is much like optimizing a transformation: \begin{align} f: X\rightarrow\hat{p} \end{align} Where X is an input vector and p̂ are the predicted probabilities (e.g. [0.20, 0.8] in a binary classification problem). But the thing is p̂ only acts as an actual probability when your model is well-calibrated.

Let's say you have pictures of cats and dogs (0 for cats and 1 for dogs) and let's also create prediction bins for different probabilities: [0.0, 0.33, 0.67, 1.0]. For a well-calibrated model, you'd expect to find 100% pictures of cats in the [0.0] bin and the same for dogs in the [1.0] bin. For the [0.33] bin, you'd expected to find 2/3 of cats and 1/3 dogs and the other way around for the [0.67] bin. If you were to find, say 9/10 cats and 1/10 dogs in the [0.33] bin then it would mean that your model is poorly calibrated.

You'll probably find a clearer explanation of it on YouTube. Overall I'd say thay you should definitely check the models's calibration curves before calibration, see if they improve after calibration and then use a binary metric of your choice (AUC, F1, etc.) to evaluate their generalization.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.