0
$\begingroup$

I'm working on a model to detect errors on human classifications.

I already have a classifier model M: X -> Y, but I need now a model M' (X,Y) -> {0,1}.

Y contains a lot of classes (~5-6k).

My main intuition is to use the model M to see what "probability" it gives to the given class y; and, if inferior to a threshold, return "False". However, I see a possible problem, where the given x is an outlier in X, and so the model M doesn't return good results. Is there particular ideas I can use for my problem ? I did not find papers working on such cases, but I'd be interested to have some.

$\endgroup$

1 Answer 1

0
$\begingroup$

simplifying the problem, solution need a multiclass-clasifier which has 5-6k classes; if the classes predicted correctly then it's "True Positive" /"True Negative" else "False Positive"/"False Negative". if no special interest in any particular class, Accuracy can be the training matrix. this is same as {0,1} for wrong/right prediction.

if you have fewer sample for some class you can assign heigher weight while training; most of the standard package have that argument.

Major Concern Area: 5-6k output classes are way too many to predict. if you have hundred of millions of data and atleast hundreds of thousands for each classs then may be Ok, but still very high chance of ending up in biased model for some classes.

  • try to bucket them and minimise as much as possible.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.