Multilabel classifcation in sklearn with soft (fuzzy) labels

Question

I have a model which is trained in sklearn on a 5-way classification problem, which performs relatively well (there are kNN and SVM versions, and both reproduce a test set with high accuracy).

When the model is applied in "real life", it is highly likely that many samples will contain linear combinations of multiple classes. So a sample may be 70% class A and 30% class B.

Much of what I have read about multilabel classification in sklearn relates to problems which don't fit this paradigm well, most of them are "tagging" type problems such as movie genre classification. Is there a way to apply my SVM/kNN models to this type of problem? I would prefer to only train on single-class examples but can modify the training set to create some multi-class samples too.

It seems I could work this by simply doing an indivdiual binary classifier for each class. However, this wouldn't give me the relative strength of each label, i.e. the linear coefficient. Is that possible?

Answer 1

For example, for a 3-class classification, we want to train with a label like $A$, which is one-hot encoded as $(1, 0, 0)$, and also with a fuzzy label like $(0.8, 0.2, 0)$. In that case, kNN and SVM of sklearn does not support fuzzy labels.

However, we can use sklearn's MultiOutputRegressor that extends a one-output Regressor such as Support Vector Regression (SVR) to multiple outputs. It is worth noting that neural networks are a natural fit for this type of label since they readily work with numerical vectors as labels.

Here is a code that goes through different types of labels for kNN, SVC (multi-class SVM), and MultiRegression SVR:

import sklearn
import pandas as pd
from sklearn.svm import SVC, SVR
from sklearn.model_selection import KFold, cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.multioutput import MultiOutputRegressor
import numpy as np

N = 1000
split = int(0.8 * N)
folds = 5
seed = 1234

# Data
np.random.seed(seed)
feature_1 = np.random.normal(0, 2, N)
feature_2 = np.random.normal(5, 6, N)
X = np.vstack([feature_1, feature_2]).T

Y_label = np.random.choice(['A', 'B', 'C'], N)

Y_one_hot = pd.get_dummies(Y_label).values

smooth_filter = np.array([0.01, 0.98, 0.01])
Y_fuzzy = np.apply_along_axis(
    lambda m: np.convolve(m, smooth_filter, mode='same'), axis=1, arr=Y_one_hot
)


kfold = KFold(n_splits=folds, random_state=seed)

kNN = KNeighborsClassifier(n_neighbors=3)
svc = SVC()
svr = SVR()
multi_svr = MultiOutputRegressor(estimator=SVR())

knn_label = np.average(cross_val_score(kNN, X, Y_label, cv=kfold))
knn_one_hot = np.average(cross_val_score(kNN, X, Y_one_hot, cv=kfold))
try:
    knn_fuzzy = np.average(cross_val_score(kNN, X, Y_fuzzy, cv=kfold))
except ValueError:
    print('kNN: fuzzy classes are not supported')
svc_label = np.average(cross_val_score(svc, X, Y_label, cv=kfold))
try:
    svc_one_hot = np.average(cross_val_score(svc, X, Y_one_hot, cv=kfold))
except ValueError:
    print('SVC: vector is not supported')
try:
    svr_one_hot = np.average(cross_val_score(svr, X, Y_one_hot, cv=kfold))
except ValueError:
    print('SVR: vector is not supported')
multi_svr_one_hot = np.average(cross_val_score(multi_svr, X, Y_one_hot, cv=kfold, scoring='neg_mean_absolute_error'))
multi_svr_fuzzy = np.average(cross_val_score(multi_svr, X, Y_fuzzy, cv=kfold, scoring='neg_mean_absolute_error'))

print('sklearn version', sklearn.__version__)
print('Y example: ',
      "label: ", Y_label[0],
      ", one hot: ", Y_one_hot[0, :],
      ", fuzzy: ", Y_fuzzy[0, :])
print('kNN label: ', knn_label)
print('kNN one hot: ', knn_one_hot)
print('SVC label: ', svc_label)
print('MultiSVR one hot: ', multi_svr_one_hot)
print('MultiSVR fuzzy: ', multi_svr_fuzzy)

Output:

kNN: fuzzy classes are not supported
SVC: vector is not supported
SVR: vector is not supported
sklearn version 0.19.1
Y example:  label:  B , one hot:  [0 1 0] , fuzzy:  [0.01 0.98 0.01]
kNN label:  0.321
kNN one hot:  0.254
SVC label:  0.332
MultiSVR one hot:  -0.4066160996805417
MultiSVR fuzzy:  -0.3970780923514713

Although kNN does not throw an exception for one-hot encoded labels, accuracy 0.254 shows that it does not work correctly with the vector.

Also, Negative Mean Absolute Error is reported for MultiSVR since the task is understood as regression. Score accuracy can only be used after changing the fuzzy labels and predictions back to a label.