3
$\begingroup$

I'm using sklearn decision trees to classify documents in two possible types "type1" and "type2".

I've isolated few features that seem pertinent and tried to combine them manually to evaluate the result of a model. When classifying the documents manually i use the following outcomes:

  • type 1
  • type 2
  • unknown

Then I give the same features to a decision tree. The results are worse in that case because it always tries to classify documents in one of the categories "type1" or "type2" but is unable to classify documents in "unknown"

Is it possible to configure a sklearn decision tree in a way that in case of high uncertainty the document won't be classified instead of selecting a category that is likely to be wrong?

$\endgroup$
  • 1
    $\begingroup$ How about teaching it to recognise type3=unknown? You might probably need to annotate some samples manually. $\endgroup$ – mapto Nov 29 '18 at 15:00
  • $\begingroup$ actually my dataset is labeled with "type1" & "type2" but in my environment we prefer to display no result instead of bad results. Of course the final goal is to have a maximum of correct "type1" & "type2". Just i wonder if a "defensive" strategy exists to avoid errors $\endgroup$ – Bertrand Nov 29 '18 at 15:07
2
$\begingroup$

TL;DR

You can train your classifier as usual and threshold on the prediction probability.

Scikit-learn

You won't get that out of the box, but you can use this simple class to do just that!

from sklearn.tree import DecisionTreeClassifier
from numpy import argmax, max

class MyClassifier(DecisionTreeClassifier):

    unknown_class = 'unknown'

    def __init__(self, no_class_threshold=0.75, **kwargs):
        self.no_class_threshold = no_class_threshold
        super().__init__(**kwargs)

    def predict(self, X):
        preds = self.predict_proba(X)
        y_pred = [self.classes_[i] if v > self.no_class_threshold else self.unknown_class
                  for i, v in zip(argmax(preds, axis=1), max(preds, axis=1))]
        return y_pred

Just use this class as you were using the DecisionTreeClassifier before. You can pass all the parameters from DecisionTreeClassifier plus an extra one: no_class_threshold.

The no_class_threshold works in the following way:

IF
    prediction_probability > no_class_threshold
THEN
    output predicted class
ELSE
    output 'unknown'

Changing classifier

You might not want to stay with the DecisionsTreeClassfier, and instead, you might want to experiment with other classifiers. You can do this by inheriting from another Scikit-learn class. Simple as that!

For example:

from sklearn.svm import SVC

class MyClassifier(SVC):
    [...]

Choosing the no_class_threshold

You can:

  • Choose it manually, to a degree that you feel confident
  • Use hyper-parameter tuning to learn it, if you have data points labelled as 'unknown'.

Example

Here is a script you can run to test all the above.

from numpy import argmax, max
from sklearn.svm import SVC


class MyClassifier(SVC):
    unknown_class = 'unknown'

    def __init__(self, no_class_threshold=0.75, **kwargs):
        self.no_class_threshold = no_class_threshold
        super().__init__(**kwargs)

    def predict(self, X):
        preds = self.predict_proba(X)
        y_pred = [self.classes_[i] if v > self.no_class_threshold else self.unknown_class
                  for i, v in zip(argmax(preds, axis=1), max(preds, axis=1))]
        return y_pred


if __name__ == '__main__':
    X = [[1, 1, 1],
         [1, 0, 1],
         [0, 0, 0],
         [0, 1, 0],
         [0, 1, 1]]
    y = ['class1', 'class1', 'class0', 'class0', 'class0']

    clf = MyClassifier(no_class_threshold=0.55, probability=True, C=1)

    clf.fit(X, y)

    X2 = [[1, 1, 0],
          [0, 1, 1],
          [0, 0, 1]]
    pred = clf.predict(X2)
    print(pred)
$\endgroup$
  • $\begingroup$ Thx for your answer Bruno, it's working as expected $\endgroup$ – Bertrand Nov 30 '18 at 16:36
  • $\begingroup$ @Bertrand I am glad to hear. Out of curiosity, which classifier did you end up using? Did you stay with a Decision Tree? $\endgroup$ – BrunoGL Nov 30 '18 at 19:30
  • $\begingroup$ i've tested both DecisionTree & SVC. The results look quite similar for now. Just to clarify, your comment says that DecisionTree classifier only returns probabilites 0 & 1. However in my test it's not the case, maybe it could updated in your answer. $\endgroup$ – Bertrand Dec 1 '18 at 13:18
  • $\begingroup$ @Bertrand thanks for the comment, I updated it. Was probably just my example ;) $\endgroup$ – BrunoGL Dec 2 '18 at 8:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.