4
$\begingroup$

I'm doing the titanic exercise on kaggle and there is a categorical Cabin attribute that has a lot of different strings: C41, C11, B20 etc. (about 100).

To be able to train my model I'm converting it to numerical attributes (using pandas get_dummies()). So in the end I get 100+ attributes.

On the test dataset however, there are less cabins, so I'll end up with fewer attributes.

I did something like this to make them equal (create columns that are in the training set and delete those that aren't):

for column in X.columns:
    if column not in X_test.columns:
        X_test[column] = 0

for column in X_test.columns:
    if column not in X.columns:
        X_test.drop([column], axis=1, inplace=True)

but I know it is not a good thing. So how else should I approach it?

I tried removing the cabin column altogether but my model performs better on test data with that column.

$\endgroup$
1
5
$\begingroup$

You could concatenate your train and test datasets, crete dummy variables and then separate them dataset.

Something like this:

train_objs_num = len(train)
dataset = pd.concat(objs=[train, test], axis=0)
dataset = pd.get_dummies(dataset)
train = copy.copy(dataset[:train_objs_num])
test = copy.copy(dataset[train_objs_num:])
$\endgroup$
2
  • $\begingroup$ @ClaudiuCreanga you might consider to chose this answer as an accepted answer. $\endgroup$ – Jekaterina Kokatjuhha Jun 4 '17 at 20:05
  • 1
    $\begingroup$ I completely disagree. You should NEVER use your test set as part of your training, Creating the dummies is part of the training pipeline. A simple example of why this doesn't work. What if now I want to make a prediction with an example that has an unknown cabin? $\endgroup$ – Valentin Calomme May 29 '20 at 10:01
6
$\begingroup$

Even though @Jekaterina Kokatjuhha's answer is accepted, I completely disagree with what it suggests. You should never make use of your test set when creating your pipeline. Technically, you don't know what your test set is until your pipeline has been completed.

Your original approach is the correct one.

To make the process less complex, I'd recommend using scikit-learn's OneHotEncoder instead of pd.get_dummies. When you fit the encoder on your training data, it will keep track of which dummies to create. In your case, this is all the cabins in your training set. This means that when you apply the encoder on your test data, it will create the same amount of columns as for your training data. If your test data contains a cabin that doesn't exist in your training data, it will simply be ignored. You also have the option to throw an error by setting the handle_unknown argument to "error"

$\endgroup$
3
$\begingroup$

As mentioned by @Valentin Calomme the right approach is to use one hot encoding method of scikit learn. The reason merging training and testing data is not beneficial this method fails in case of when you put model into production & then suddenly mapping changes & ultimately leads to failure of the model.

Follow below implementation for end to end implementation of that method:

import pandas as pd
import pickle
from sklearn.preprocessing import OneHotEncoding

df_trn = pd.read_csv('train.csv')

# You can use other methods to remove nan records
df_trn.dropna(axis=0, inplace=True)

# Always perform following step on training records
cabin_onehot = OneHotEncoder()
cabin_arr = cabin_onehot.fit_transform(df_trn.loc[:, ['Cabin']]).toarray()

# Save this onehot encoding object for reuse purpose
with open('cb.pkl', 'wb') as f:
    pickle.dump(cabin_onehot, f)

# Write code to merge df_trn & cabin_arr
...

# Now let's apply same technique on test records
df_test = df.read_csv('test.csv')

# Load it before using it & same thing we do we when use in production/testing environment
with open('cb.pkl', 'rb') as f:
    cabin_onehot = pickle.load(f)
cabin_arr_test = cabin_onehot.transform(df_test.loc[:, ['Cabin']]).toarray()

In this way you will not face issue of different dimension in test records.

$\endgroup$
-1
$\begingroup$

You would prefer to concatenate data first and then convert in dummies followed by again splitting them in Training and Testing dataset as suggested by @Jekaterina Kokatjuhha.

Though the option of the merging of the dataset could be problem-specific, you may also read the article here for getting more understanding about having different observations within Train and Test Dataset.

$\endgroup$
-1
$\begingroup$

Different categories in the train and test set is a massive problem that ideally won't occur if you do the train test split properly.

Consider using something called as Stratified Shuffle Split and then one-hot-encode the data before modelling further. Stratified Shuffle Split essentially preserves the percentage of the categorical features in each column and then makes the split. This leads to well-balanced Training and Testing data-sets.

Regards,

$\endgroup$
1
  • $\begingroup$ Stratified shuffle should be done at the class level. Doing it at the attribute level is not a good idea as if you stratify for one feature, you may not stratify for another. $\endgroup$ – Valentin Calomme May 29 '20 at 10:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.