Error in using fit() on RandomForest Classifier where X was a pandas.DataFRame object

Question

On using fit() method on sklearn.ensemble.RandomForestClassifier I am getting a value error that says.

ValueError: could not convert string to float: 'male'

The data-set used is the one in Titanic:Machine Learning from Disaster competition on Kaggle. Here is the link- https://www.kaggle.com/c/titanic Can someone please help me how to deal with this, why is it occurring and how to prevent it in future.

Note-There are no NaN in my DataFrame for train_X, i.e I have replaced all NaN with df.fillna(df.mean()), also I cross-checked that no NaN values exist by using

train_X.isnull().sum()

where, train_X is the training data for features. Please Help!!

Answer 1

As an extension to @marco_gorelli's answer, another option apart from one-hot encoding is to use LabelEncoder from sklearn.

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()
df['sex_enc'] = le.fit_transform(df['sex'])
df['sex_enc'] = df['sex_enc'].astype('category')

Answer 2

You can't pass a categorical variable as it is to one of sklearn's classifiers. One approach for dealing with this is to dummy-encode the column in question.

I realise that it's not a strict requirement to post minimal, complete, verifiable code on this site, but it will help if you provide code that can be run by others so that they can go straight to the solution without having to guess what's happening from your error message.

Here's some code which reproduces your error:

import pandas as pd
from sklearn.ensemble import RandomForestClassifier

df = pd.DataFrame({'sex': ['male', 'female', 'female', 'male', 'female'], 'survived': [0, 1, 1, 0, 1]})
rf = RandomForestClassifier()
rf.fit(df.drop('survived', axis=1), df['survived'])

We can fix the error by using the get_dummies function from pandas. The following code gives no errors:

df_dummies = pd.get_dummies(df)
rf.fit(df_dummies.drop('survived', axis=1), df_dummies['survived'])