I'm doing the titanic exercise on kaggle and there is a categorical Cabin attribute that has a lot of different strings: C41, C11, B20 etc. (about 100).
To be able to train my model I'm converting it to numerical attributes (using pandas get_dummies()). So in the end I get 100+ attributes.
On the test dataset however, there are less cabins, so I'll end up with fewer attributes.
I did something like this to make them equal (create columns that are in the training set and delete those that aren't):
for column in X.columns: if column not in X_test.columns: X_test[column] = 0 for column in X_test.columns: if column not in X.columns: X_test.drop([column], axis=1, inplace=True)
but I know it is not a good thing. So how else should I approach it?
I tried removing the cabin column altogether but my model performs better on test data with that column.