I am using the XGBoost for classification of text data. There are the 3 different classes in the training dataset.

classifier = Pipeline([
    ('features', FeatureUnion([
        ('text', Pipeline([
            ('colext', TextSelector('Issue')),
            ('tfidf', TfidfVectorizer(tokenizer=Tokenizer, stop_words='english',
                     min_df=.0025, max_df=0.25, ngram_range=(1,3))),
            ('svd', TruncatedSVD(algorithm='randomized', n_components=300)), #for XGB
    ('clf', XGBClassifier(n_estimators=300,max_depth=3, learning_rate=0.1)),
#    ('clf', RandomForestClassifier()),

As per the classification results, the class for which prediction probability is highest is assigned to the data point. For example, if the prediction probability for class A is .67, then that data point is assigned to that category(Class A).


But the requirement is to assign the data point to the 4th Category that is "UnDetermined" if the prediction probability for the data point does not differ much among the classes. For example, if the prediction probability of the datapoint for three classes is .32,.33,.35, then can we mark it as Undetermined. So that the user can review the undetermined category and assign that to the appropriate class.

But I am not sure how to set the cutoff probability for multiclass classification problem? PLease let me know how to identify the cutoff probability. So that i can mark the data points with prediction probability less than cutoff probability as "Undetermined"


You don't set it in xgboost. Its job is to return probabilities in predict_proba. predict does the logical thing and tells you the most likely class.

If you want to interpret the probabilities differently, you'd have to write code to do so. It depends on what "does not much differ" means. For example if you simply mean "the most likely class is has probability < 0.5", in numpy it's something like np.argwhere(probabilities.max(axis=1) < 0.5). Those rows are what are undetermined. For the rest the prediction is the argmax of the row.

Of course, that's not the only definition of when it's undetermined, so you need to figure out what the rule is that you're meant to implement. You could base it on entropy too, not simply the max class probability.

  • $\begingroup$ Is there is a statistical way of defining the undetermined category or I can define my own probability cutoff? $\endgroup$
    – user88518
    Feb 13 '20 at 4:29
  • $\begingroup$ Have a look at the 'uncertainty sampling' in modAL: the docs describe a few different definitions. Which one is best depends on your goal. modal-python.readthedocs.io/en/latest/content/query_strategies/… $\endgroup$
    – Sean Owen
    Feb 13 '20 at 17:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.