Faced problem while applying OneHotEncoder

Question

For classification, I was trying to convert categorical data into numeric by applying OneHotEncoder. But it shows error could not convert string to float

Here is the sample of my categorical data set and code of One Hot Encoding.

# TODO: create a OneHotEncoder object, and fit it to all of X from sklearn.preprocessing import OneHotEncoder

# 1. INSTANTIATE enc = OneHotEncoder()

# 2. FIT enc.fit(train_obj)

# 3. Transform train_ = enc.transform(train_obj) train_.head()

And error message:

I couldn't understand what's the problem & how could I solve it.

If I apply get_dummies() method, is it similar of OneHotEncoder ?

Answer 1

Consider the following demo:

Source DF:

In [53]: df
Out[53]:
   A          B                   C
0  1  Bachelors  Married-civ-spouse
1  2    HS-grad            Divorsed
2  3       11th  Married-civ-spouse

Option 1:

In [54]: for c in df.columns[df.dtypes.eq('object')]:
    ...:     df = df.join(df[c].str.get_dummies())
    ...:

Result:

In [55]: df
Out[55]:
   A          B                   C  11th  Bachelors  HS-grad  Divorsed  Married-civ-spouse
0  1  Bachelors  Married-civ-spouse     0          1        0         0                   1
1  2    HS-grad            Divorsed     0          0        1         1                   0
2  3       11th  Married-civ-spouse     1          0        0         0                   1

Option 2:

In [58]: df = df.join(pd.get_dummies(df.select_dtypes(['object'])))

In [59]: df
Out[59]:
   A          B                   C  B_11th  B_Bachelors  B_HS-grad  C_Divorsed  C_Married-civ-spouse
0  1  Bachelors  Married-civ-spouse       0            1          0           0                     1
1  2    HS-grad            Divorsed       0            0          1           1                     0
2  3       11th  Married-civ-spouse       1            0          0           0                     1

Answer 2

To get from a string to one hot encoding is basically a 2-step process.

sklearn's OneHotEncoder function:

Encode[s] categorical integer features using a one-hot aka one-of-K scheme. (see here)

So, one first has to convert strings to integers, which can be easily done using sklearn's LabelEncoder:

Encode labels with value between 0 and n_classes-1. (see here)

There several such examples on StackOverflow, e.g. here.