Why do probabilities sum to one and how can I set optimal threshold level?

Question

I am working on a text classification use case. The training data has two classes, so the XBBoostClassifier and onevsrest model is classifying the test data into either of the two classes. But my requirement is to classify either into given the classes or if no match found, then set it as 'undetermined" so that i can manually classify the data.

I tried the following OneVsRest Classifier

pl = Pipeline([
        ('vec', CountVectorizer(token_pattern = tks)),
        ('clf', OneVsRestClassifier(LogisticRegression()))
    ])
pl.fit(x,y)
predictions = pl.predict_proba(test.comment_text)

But the sum of the probabilities is one and moreover the probability is above 90 for the class to which the data belongs to.

Please clarify the following points 1. Why the probability is always one? whether it means that the data is mutually exclusive? 2. The probabilities are like this

CLASS 1           :CLASS 2
0.892993358265023 : 0.106808845640795
0.999999742528922 : 2.57685096542208E-07 

Does that mean that in first case, probability is only 90% for class 1 and hence the classifier is not able to classify the data properly. However in other case, there is clear cut difference as the probability is around 99%

Can I set the threshold, like 90%, and conclude if the probability is less than 90% , let the users manually classify the data?

Please provide your inputs

Answer 1

Question 1. Why do probabilities sum to 1: Probability theory. Probabilities sum 1 because that’s how we define them. It just so happens that, by forcing them to sum 1, they have an intuitive interpretation and also calculations end up being easier. But this is mere convenience. Probabilities (or, more specifically, probability measures) could have been defined to sum 12 or 100 or whatever number you prefer. It doesn’t really matter.

Further sub-questions:

• Its not the data thats exclusive its the classes.
• Classifier is able to classify just with lower probability ( I would actually argue that 99999% is to much maybe he is overfitting)

Question 2 Can I set the threshold, like 90%, and conclude if the probability is less than 90% , let the users manually classify the data? Let the data tell you optimal cut-off level/threshold (from the ROC curve, read it up)

def optimal_cutoff(ground_truths: np.array,
                             predictions: np.array) -> float:
        """

        :param ground_truths: array of arrays (matrix) of all ground truths
        :param predictions: array of arrays (matrix) of all predictions
        :return: optimal cut-off level
        """
        optimal_thresholds = []
        for y_train, y_pred in zip(ground_truths, predictions):
            fpr, tpr, threshold = roc_curve(y_train, y_pred)
            optimal_idx = np.argmax(tpr - fpr)
            optimal_thresholds.append(threshold[optimal_idx])

        return sum(optimal_thresholds) / (len(optimal_thresholds))

So you would calculate cut-off on your available GROUND truths. Now if, when you specified optimal cut-off level your metric (lets say f1) falls behind certain level you can tell users to check it up.