Scikit Learn OneHotEncoded Features causing error in classifier

Question

I’m trying to prepare data for input to a Decision Tree and Multinomial Naïve Bayes Classifier.

This is what my data looks like (pandas dataframe):

Label  Feat1  Feat2  Feat3  Feat4

0        1     3       2      1
1        0     1       1      2
2        2     2       1      1
3        3     3       2      3

I have split the data into dataLabel and dataFeatures. Prepared dataLabel using dataLabel.ravel()

I need to discretize features so the classifiers treat them as being categorical not numerical.

I’m trying to do this using OneHotEncoder:

enc = OneHotEncoder()

enc.fit(dataFeatures)
chk = enc.transform(dataFeatures)
from sklearn.naive_bayes import MultinomialNB

mnb = MultinomialNB()

from sklearn import metrics
from sklearn.cross_validation import cross_val_score
scores = cross_val_score(mnb, Y, chk, cv=10, scoring='accuracy')

I get this error: bad input shape (64, 16)

This is the shape of label and input:

dataLabel.shape = 72 chk.shape = 72,16

Why won't the classifier accept the onehotencoded features?

EDIT: Adding how I got dataFeatures

dataFeatures = data[['Accpred', 'Gyrpred', 'Barpred', 'altpred']]

Y = dataLabel.ravel()

Answer 1

scores = cross_val_score(mnb, Y, chk, cv=10, scoring='accuracy')

You have your Y and chk switched. That's it. :)

The signature of cross_val_score is sklearn.cross_validation.cross_val_score(estimator, X, y).

X is a matrix and y is a 1D vector with your class labels.

unlike in R, most (or all?) sklearn models do not support categorical variables. Most of the time, encoding your feature matrix X into what is called one-hot encoding is good enough.

Notice that, in some models, this hack is not the same as true native categorical support, and the performance of the model will be worse.

Invert One-Hot Encoding

Sklearn does not seem to have an easy method to invert the one-hot encoding.

It is not trivial how to do this. I found this suggestion:

def inverse(enc, out, shape):
    return np.array([enc.active_features_[col] for col in out.sorted_indices().indices]).reshape(shape) - enc.feature_indices_[:-1]

Example:

import numpy as np
from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder()
X = np.array([[0, 0, 3], [1, 1, 0], [0, 2, 1], [1, 0, 2]]))
Z = enc.fit_transform(X)
print(inverse(enc, Z, X.shape))
# [[0 0 3]
#  [1 1 0]
#  [0 2 1]
#  [1 0 2]]
print(X)
# [[0 0 3]
#  [1 1 0]
#  [0 2 1]
#  [1 0 2]]

Notice:

• This only works when HotOneEncoding(sparse=True) (default) because it uses scipy sparse matrix methods (this could be changed by making the code only use numpy methods), but this is probably what you want since working with a dense matrix will kill your memory anyhow
• I think this will only work if your variables are within the range [0,something] because you lose that information in the transformation (no work-around for this other than you using something like DictVectorizer which offers you more control over the transformation.