# Classify samples based on other sample probabilities

I was wondering if there's a way to train a classifier or set up a way of classifying after that can classify certain samples as some relationship between the previous two.

I notice that, for example, when I use the predict_proba from scikit-learn's RandomForest, I can see the probability that a class was predicted like this: [0.3, 0.43, 0.27]. I want to do something like: if | p(class1) - p(class2) | < ε then class 3 should be boosted. Maybe this means using a binary classifier at first and then checking the relationship between the binary classes.

To describe the situation further, class 1 and class 2 are distinct and class 3 has a mixture of both features.

• do you have labels for class 3 (or can derive them from 1,2). You could expand the classes like, e.g 1 true and 3 true, 1 false and 3 true etc. Should give you n classes. Then you can train as usual with n classes. – Peter May 30 '19 at 10:43
• i have labels for class 3, and i want to factor in both the combination of class 1/2 and 3 – frei Jun 1 '19 at 7:58

This is a case of multilabel/multioutput classification. You have a corpus of data in which several classes can be true for a single sample. Moreover - where one class is literally a mixture of the other two classes. A much more common problem than a lot of us would wish it to be.

Note: I'll rename classes 1, 2 and three into classes 0, 1 and 2 respectively; since that is how sklearn enumerates them.

The sklearn's RandomForest supports multilabel classification out of the box, therefore instead of organizing your data as follows:

X                   | y
feature1  feature2  | label
--------------------+------
0.1       0.3       | 0
0.2       0.1       | 1
0.7       0.5       | 1
0.8       0.3       | 1
0.6       0.6       | 1 (but also 0 - so probably should be 0 and 1 - class 2?)
0.3       0.9       | 0
0.5       0.5       | 0 (but also 1 - so probably should be both as well- class 2?)


Organize the data in the following way:

X                   | Y
feature1  feature2  | class0?  class1?
--------------------+-----------------
0.1       0.3       | 1        0
0.2       0.1       | 0        1
0.7       0.5       | 0        1
0.8       0.3       | 0        1
0.6       0.6       | 1        1
0.3       0.9       | 1        0
0.5       0.5       | 1        1


In other words, make your label vector into a matrix - i.e. both $$X$$ and $$Y$$ will have two dimensions now. sklearn's RandomForest will accept that inside it's fit() and inside it's predict() methods (and inside predict_proba() as well).

The only tricky bit may be the interpretation of the output of predict_proba() in multilabel/multioutput classification, for example (watch for typos, I'm doing this code from memory):

import numpy as np
from sklearn.ensemble import RandomForestClassifier

X = np.random.random((3, 3))
Y = np.array([[0, 1],
[1, 0],
[1, 1]])
model = RandomForestClassifier()
model.fit(X, Y)
model.predict(X)

np.array([[0., 1.],
[1., 0.],
[1., 1.]])

model.predict_proba(X)

[np.array([[0.6, 0.4],
[0.7, 0.3],
[0.1, 0.9]]),
np.array([[0.9, 0.1],
[1.,  0. ],
[0.2, 0.8]])]


In summary, predict_proba did return a list of two elements: the first element is the probability of class 0 independently of class 1, whilst the second element in the list is the probability of class 1 independently of class 0. Whether the is a high probability of class 0 and a high probability of class 1 then you have a prediction of [1, 1].

• very detailed answer, thank you for the explanation! right now my classes are actually already labelled as 0-1-2 and when labeling i used 2 to indicate that there was a mixture of both classes – frei Jun 1 '19 at 7:57
• Hi, when you said the first element is the probability of class 0 independently of class 1, whilst the second element in the list is the probability of class 1 independently of class 1, did you mean to say probability of class 1 independently of class 0? – frei Jun 3 '19 at 1:37
• @frei - Yep, that's what I meant but wrote something different. Now fixed. It was a good catch, thanks for that. – grochmal Jun 3 '19 at 1:59
• Hi @grochmal, sorry to bother you again but I was wondering - if instead of labeling according to this rubrick I instead did a post-processing step on a binary classifier where I checked the predict_proba results and did a calculation there, would that yield very different results than training on multiclass? Given that the return is the probability of classes independent of each other, would training them as just binary end up being similar? – frei Jun 5 '19 at 8:54
• @frei - Very different results, probably not. Different results, yes. Using classifiers that cannot do multioutput (e.g. SVMs) doing two binary classifiers and comparing probabilities would be your only option but with other classifiers you do have more options. The idea of multioutput is that, if the theory (of the problem we are trying to solve) suggests three classes (one a combination) then we have three dense regions in space. With multiclass/multioutput we do the decision line between the dense regions, with two binary classifiers we divide through the middle of one of the regions. – grochmal Jun 5 '19 at 11:55