Different number of features in train vs test

Question

I'm doing the titanic exercise on kaggle and there is a categorical Cabin attribute that has a lot of different strings: C41, C11, B20 etc. (about 100).

To be able to train my model I'm converting it to numerical attributes (using pandas get_dummies()). So in the end I get 100+ attributes.

On the test dataset however, there are less cabins, so I'll end up with fewer attributes.

I did something like this to make them equal (create columns that are in the training set and delete those that aren't):

for column in X.columns:
    if column not in X_test.columns:
        X_test[column] = 0

for column in X_test.columns:
    if column not in X.columns:
        X_test.drop([column], axis=1, inplace=True)

but I know it is not a good thing. So how else should I approach it?

I tried removing the cabin column altogether but my model performs better on test data with that column.

Answer 1

You could concatenate your train and test datasets, crete dummy variables and then separate them dataset.

Something like this:

train_objs_num = len(train)
dataset = pd.concat(objs=[train, test], axis=0)
dataset = pd.get_dummies(dataset)
train = copy.copy(dataset[:train_objs_num])
test = copy.copy(dataset[train_objs_num:])

Answer 2

Even though @Jekaterina Kokatjuhha's answer is accepted, I completely disagree with what it suggests. You should never make use of your test set when creating your pipeline. Technically, you don't know what your test set is until your pipeline has been completed.

Your original approach is the correct one.

To make the process less complex, I'd recommend using scikit-learn's OneHotEncoder instead of pd.get_dummies. When you fit the encoder on your training data, it will keep track of which dummies to create. In your case, this is all the cabins in your training set. This means that when you apply the encoder on your test data, it will create the same amount of columns as for your training data. If your test data contains a cabin that doesn't exist in your training data, it will simply be ignored. You also have the option to throw an error by setting the handle_unknown argument to "error"

Answer 3

As mentioned by @Valentin Calomme the right approach is to use one hot encoding method of scikit learn. The reason merging training and testing data is not beneficial this method fails in case of when you put model into production & then suddenly mapping changes & ultimately leads to failure of the model.

Follow below implementation for end to end implementation of that method:

import pandas as pd
import pickle
from sklearn.preprocessing import OneHotEncoding

df_trn = pd.read_csv('train.csv')

# You can use other methods to remove nan records
df_trn.dropna(axis=0, inplace=True)

# Always perform following step on training records
cabin_onehot = OneHotEncoder()
cabin_arr = cabin_onehot.fit_transform(df_trn.loc[:, ['Cabin']]).toarray()

# Save this onehot encoding object for reuse purpose
with open('cb.pkl', 'wb') as f:
    pickle.dump(cabin_onehot, f)

# Write code to merge df_trn & cabin_arr
...

# Now let's apply same technique on test records
df_test = df.read_csv('test.csv')

# Load it before using it & same thing we do we when use in production/testing environment
with open('cb.pkl', 'rb') as f:
    cabin_onehot = pickle.load(f)
cabin_arr_test = cabin_onehot.transform(df_test.loc[:, ['Cabin']]).toarray()

In this way you will not face issue of different dimension in test records.

Answer 4

Hope this works. If the models trained are GLM,DT,RF you can extract the train data column names using the below syntax

train_data<-attr(model$terms, 'term.labels')

  df<-as.data.frame(train_data)

  df<-as.data.frame(do.call(rbind,df))

  names(df) <- df[1,]

  df <- df[-1,]

Now,convert categorical columns to dummy variables in the test dataset. The below line of code will make the train and test data column names same and set train_data column names to 0 which are not present in the test dataset.

test_data[setdiff(names(df), names(test_data))] <- 0

Answer 5

You would prefer to concatenate data first and then convert in dummies followed by again splitting them in Training and Testing dataset as suggested by @Jekaterina Kokatjuhha.

Though the option of the merging of the dataset could be problem-specific, you may also read the article here for getting more understanding about having different observations within Train and Test Dataset.

Answer 6

Different categories in the train and test set is a massive problem that ideally won't occur if you do the train test split properly.

Consider using something called as Stratified Shuffle Split and then one-hot-encode the data before modelling further. Stratified Shuffle Split essentially preserves the percentage of the categorical features in each column and then makes the split. This leads to well-balanced Training and Testing data-sets.

Regards,