I am using the XGBoost for classification of text data. There are the 3 different classes in the training dataset.
classifier = Pipeline([ ('features', FeatureUnion([ ('text', Pipeline([ ('colext', TextSelector('Issue')), ('tfidf', TfidfVectorizer(tokenizer=Tokenizer, stop_words='english', min_df=.0025, max_df=0.25, ngram_range=(1,3))), ('svd', TruncatedSVD(algorithm='randomized', n_components=300)), #for XGB ])) ])), ('clf', XGBClassifier(n_estimators=300,max_depth=3, learning_rate=0.1)), # ('clf', RandomForestClassifier()), ])
As per the classification results, the class for which prediction probability is highest is assigned to the data point. For example, if the prediction probability for class A is .67, then that data point is assigned to that category(Class A).
But the requirement is to assign the data point to the 4th Category that is "UnDetermined" if the prediction probability for the data point does not differ much among the classes. For example, if the prediction probability of the datapoint for three classes is .32,.33,.35, then can we mark it as Undetermined. So that the user can review the undetermined category and assign that to the appropriate class.
But I am not sure how to set the cutoff probability for multiclass classification problem? PLease let me know how to identify the cutoff probability. So that i can mark the data points with prediction probability less than cutoff probability as "Undetermined"