For example, for a 3-class classification, we want to train with a label like $A$, which is one-hot encoded as $(1, 0, 0)$, and also with a fuzzy label like $(0.8, 0.2, 0)$. In that case, kNN and SVM of sklearn does not support fuzzy labels.
However, we can use sklearn's MultiOutputRegressor
that extends a one-output Regressor such as Support Vector Regression (SVR) to multiple outputs. It is worth noting that neural networks are a natural fit for this type of label since they readily work with numerical vectors as labels.
Here is a code that goes through different types of labels for kNN, SVC (multi-class SVM), and MultiRegression SVR:
import sklearn
import pandas as pd
from sklearn.svm import SVC, SVR
from sklearn.model_selection import KFold, cross_val_score
from sklearn.neighbors import KNeighborsClassifier
from sklearn.multioutput import MultiOutputRegressor
import numpy as np
N = 1000
split = int(0.8 * N)
folds = 5
seed = 1234
# Data
np.random.seed(seed)
feature_1 = np.random.normal(0, 2, N)
feature_2 = np.random.normal(5, 6, N)
X = np.vstack([feature_1, feature_2]).T
Y_label = np.random.choice(['A', 'B', 'C'], N)
Y_one_hot = pd.get_dummies(Y_label).values
smooth_filter = np.array([0.01, 0.98, 0.01])
Y_fuzzy = np.apply_along_axis(
lambda m: np.convolve(m, smooth_filter, mode='same'), axis=1, arr=Y_one_hot
)
kfold = KFold(n_splits=folds, random_state=seed)
kNN = KNeighborsClassifier(n_neighbors=3)
svc = SVC()
svr = SVR()
multi_svr = MultiOutputRegressor(estimator=SVR())
knn_label = np.average(cross_val_score(kNN, X, Y_label, cv=kfold))
knn_one_hot = np.average(cross_val_score(kNN, X, Y_one_hot, cv=kfold))
try:
knn_fuzzy = np.average(cross_val_score(kNN, X, Y_fuzzy, cv=kfold))
except ValueError:
print('kNN: fuzzy classes are not supported')
svc_label = np.average(cross_val_score(svc, X, Y_label, cv=kfold))
try:
svc_one_hot = np.average(cross_val_score(svc, X, Y_one_hot, cv=kfold))
except ValueError:
print('SVC: vector is not supported')
try:
svr_one_hot = np.average(cross_val_score(svr, X, Y_one_hot, cv=kfold))
except ValueError:
print('SVR: vector is not supported')
multi_svr_one_hot = np.average(cross_val_score(multi_svr, X, Y_one_hot, cv=kfold, scoring='neg_mean_absolute_error'))
multi_svr_fuzzy = np.average(cross_val_score(multi_svr, X, Y_fuzzy, cv=kfold, scoring='neg_mean_absolute_error'))
print('sklearn version', sklearn.__version__)
print('Y example: ',
"label: ", Y_label[0],
", one hot: ", Y_one_hot[0, :],
", fuzzy: ", Y_fuzzy[0, :])
print('kNN label: ', knn_label)
print('kNN one hot: ', knn_one_hot)
print('SVC label: ', svc_label)
print('MultiSVR one hot: ', multi_svr_one_hot)
print('MultiSVR fuzzy: ', multi_svr_fuzzy)
Output:
kNN: fuzzy classes are not supported
SVC: vector is not supported
SVR: vector is not supported
sklearn version 0.19.1
Y example: label: B , one hot: [0 1 0] , fuzzy: [0.01 0.98 0.01]
kNN label: 0.321
kNN one hot: 0.254
SVC label: 0.332
MultiSVR one hot: -0.4066160996805417
MultiSVR fuzzy: -0.3970780923514713
Although kNN does not throw an exception for one-hot encoded labels, accuracy 0.254
shows that it does not work correctly with the vector.
Also, Negative Mean Absolute Error is reported for MultiSVR since the task is understood as regression. Score accuracy
can only be used after changing the fuzzy labels and predictions back to a label.