0
$\begingroup$

When assessing a binary classification task, it is possible to search for particular threshold in order to have better score on some metrics (f1,recall,etc) through numerous methods. Unfortunately, it looks like the method cannot be applied on categorical classification (more than two classes) task. I've thought about training a Simple Classifier (SVC,Log,...,Tree) on top of a already trained model in order to find best thresholds to apply on outputs to maximize similarity of results.

My proposed workflow is to train a model on training set and evaluate it, then use outputs on training set of this model as entry for another model which may learn how to handle threshold to maximize similarity to ground truth on training set (all of those steps being validated using the same strategy).

I've never heard about it and I want to know whether it makes sense ?

$\endgroup$
1
$\begingroup$

What you are talking about sounds a lot like meta-learning (use outputs of several models and use another model to train on that predictions). You can read through more detailed example here (link to machinelearningmastery)

Should you do it for your binary classification task - no. (this in not a strict "no", but generally you should not do it given your task can really benefit from such approach, but this has little to do with classification threshold)

About threshold search, you can check this thread with quite good answers: https://stats.stackexchange.com/a/386433/232020

From your question, it sounds like you are dealing with the binary dataset, not sure what you mean by Unfortunately, the method seems not to be scaled on categorical classification.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.