When assessing a binary classification task, it is possible to search for particular threshold in order to have better score on some metrics (f1,recall,etc) through numerous methods. Unfortunately, it looks like the method cannot be applied on categorical classification (more than two classes) task. I've thought about training a Simple Classifier (SVC,Log,...,Tree) on top of a already trained model in order to find best thresholds to apply on outputs to maximize similarity of results.
My proposed workflow is to train a model on training set and evaluate it, then use outputs on training set of this model as entry for another model which may learn how to handle threshold to maximize similarity to ground truth on training set (all of those steps being validated using the same strategy).
I've never heard about it and I want to know whether it makes sense ?