This is a case of multilabel/multioutput classification. You have a corpus of data in which several classes can be true for a single sample. Moreover - where one class is literally a mixture of the other two classes. A much more common problem than a lot of us would wish it to be.
Note: I'll rename classes 1, 2 and three into classes 0, 1 and 2 respectively; since that is how sklearn
enumerates them.
The sklearn
's RandomForest
supports multilabel classification out of the box, therefore instead of organizing your data as follows:
X | y
feature1 feature2 | label
--------------------+------
0.1 0.3 | 0
0.2 0.1 | 1
0.7 0.5 | 1
0.8 0.3 | 1
0.6 0.6 | 1 (but also 0 - so probably should be 0 and 1 - class 2?)
0.3 0.9 | 0
0.5 0.5 | 0 (but also 1 - so probably should be both as well- class 2?)
Organize the data in the following way:
X | Y
feature1 feature2 | class0? class1?
--------------------+-----------------
0.1 0.3 | 1 0
0.2 0.1 | 0 1
0.7 0.5 | 0 1
0.8 0.3 | 0 1
0.6 0.6 | 1 1
0.3 0.9 | 1 0
0.5 0.5 | 1 1
In other words, make your label vector into a matrix - i.e. both $X$ and $Y$ will have two dimensions now. sklearn
's RandomForest
will accept that inside it's fit()
and inside it's predict()
methods (and inside predict_proba()
as well).
The only tricky bit may be the interpretation of the output of predict_proba()
in multilabel/multioutput classification, for example (watch for typos, I'm doing this code from memory):
import numpy as np
from sklearn.ensemble import RandomForestClassifier
X = np.random.random((3, 3))
Y = np.array([[0, 1],
[1, 0],
[1, 1]])
model = RandomForestClassifier()
model.fit(X, Y)
model.predict(X)
np.array([[0., 1.],
[1., 0.],
[1., 1.]])
model.predict_proba(X)
[np.array([[0.6, 0.4],
[0.7, 0.3],
[0.1, 0.9]]),
np.array([[0.9, 0.1],
[1., 0. ],
[0.2, 0.8]])]
In summary, predict_proba
did return a list of two elements: the first element is the probability of class 0 independently of class 1, whilst the second element in the list is the probability of class 1 independently of class 0. Whether the is a high probability of class 0 and a high probability of class 1 then you have a prediction of [1, 1]
.