Classify samples based on other sample probabilities

Question

I was wondering if there's a way to train a classifier or set up a way of classifying after that can classify certain samples as some relationship between the previous two.

I notice that, for example, when I use the predict_proba from scikit-learn's RandomForest, I can see the probability that a class was predicted like this: [0.3, 0.43, 0.27]. I want to do something like: if | p(class1) - p(class2) | < ε then class 3 should be boosted. Maybe this means using a binary classifier at first and then checking the relationship between the binary classes.

To describe the situation further, class 1 and class 2 are distinct and class 3 has a mixture of both features.

Answer 1

This is a case of multilabel/multioutput classification. You have a corpus of data in which several classes can be true for a single sample. Moreover - where one class is literally a mixture of the other two classes. A much more common problem than a lot of us would wish it to be.

Note: I'll rename classes 1, 2 and three into classes 0, 1 and 2 respectively; since that is how sklearn enumerates them.

The sklearn's RandomForest supports multilabel classification out of the box, therefore instead of organizing your data as follows:

X                   | y
feature1  feature2  | label
--------------------+------
0.1       0.3       | 0
0.2       0.1       | 1
0.7       0.5       | 1
0.8       0.3       | 1
0.6       0.6       | 1 (but also 0 - so probably should be 0 and 1 - class 2?)
0.3       0.9       | 0
0.5       0.5       | 0 (but also 1 - so probably should be both as well- class 2?)

Organize the data in the following way:

X                   | Y
feature1  feature2  | class0?  class1?
--------------------+-----------------
0.1       0.3       | 1        0
0.2       0.1       | 0        1
0.7       0.5       | 0        1
0.8       0.3       | 0        1
0.6       0.6       | 1        1
0.3       0.9       | 1        0
0.5       0.5       | 1        1

In other words, make your label vector into a matrix - i.e. both $X$ and $Y$ will have two dimensions now. sklearn's RandomForest will accept that inside it's fit() and inside it's predict() methods (and inside predict_proba() as well).

The only tricky bit may be the interpretation of the output of predict_proba() in multilabel/multioutput classification, for example (watch for typos, I'm doing this code from memory):

import numpy as np
from sklearn.ensemble import RandomForestClassifier

X = np.random.random((3, 3))
Y = np.array([[0, 1],
              [1, 0],
              [1, 1]])
model = RandomForestClassifier()
model.fit(X, Y)
model.predict(X)

    np.array([[0., 1.],
              [1., 0.],
              [1., 1.]])

model.predict_proba(X)

    [np.array([[0.6, 0.4],
               [0.7, 0.3],
               [0.1, 0.9]]),
     np.array([[0.9, 0.1],
               [1.,  0. ],
               [0.2, 0.8]])]

In summary, predict_proba did return a list of two elements: the first element is the probability of class 0 independently of class 1, whilst the second element in the list is the probability of class 1 independently of class 0. Whether the is a high probability of class 0 and a high probability of class 1 then you have a prediction of [1, 1].