XGBoost: How to set the probability threshold for multi class classification

Question

I am using the XGBoost for classification of text data. There are the 3 different classes in the training dataset.

classifier = Pipeline([
    ('features', FeatureUnion([
        ('text', Pipeline([
            ('colext', TextSelector('Issue')),
            ('tfidf', TfidfVectorizer(tokenizer=Tokenizer, stop_words='english',
                     min_df=.0025, max_df=0.25, ngram_range=(1,3))),
            ('svd', TruncatedSVD(algorithm='randomized', n_components=300)), #for XGB
        ]))
    ])),
    ('clf', XGBClassifier(n_estimators=300,max_depth=3, learning_rate=0.1)),
#    ('clf', RandomForestClassifier()),
    ])

As per the classification results, the class for which prediction probability is highest is assigned to the data point. For example, if the prediction probability for class A is .67, then that data point is assigned to that category(Class A).

predictionProbability=classifier.predict_proba(X_test)

But the requirement is to assign the data point to the 4th Category that is "UnDetermined" if the prediction probability for the data point does not differ much among the classes. For example, if the prediction probability of the datapoint for three classes is .32,.33,.35, then can we mark it as Undetermined. So that the user can review the undetermined category and assign that to the appropriate class.

But I am not sure how to set the cutoff probability for multiclass classification problem? PLease let me know how to identify the cutoff probability. So that i can mark the data points with prediction probability less than cutoff probability as "Undetermined"

Answer 1

You don't set it in xgboost. Its job is to return probabilities in predict_proba. predict does the logical thing and tells you the most likely class.

If you want to interpret the probabilities differently, you'd have to write code to do so. It depends on what "does not much differ" means. For example if you simply mean "the most likely class is has probability < 0.5", in numpy it's something like np.argwhere(probabilities.max(axis=1) < 0.5). Those rows are what are undetermined. For the rest the prediction is the argmax of the row.

Of course, that's not the only definition of when it's undetermined, so you need to figure out what the rule is that you're meant to implement. You could base it on entropy too, not simply the max class probability.