How to deal with Different Shapes of X_train and X_test after OneHotEncoding?

Question

I am trying to perform OneHotEncoding as well as feature scaling on my training and testing data separately, steps I did:

X = df.drop("target", axis = 1)
y = df.target

X_train,X_test,y_train,y_test = train_test_split(X, y, test_size=0.2)

categorial_features = [List of Categorical Features]
numerical_features = [List of Numerical Features]

one_hot = OneHotEncoder()
scaler = StandardScaler()
tranformer = ColumnTransformer([("one_hot", one_hot, categorial_features),("standard_scaler", scaler, numerical_features)], remainder = "passthrough")

transformed_X_train = tranformer.fit_transform(X_train)
transformed_X_test = tranformer.fit_transform(X_test)

But now the shapes of transformed_X_train, and transformed_X_test are different, I know the reason why it is different, but I want to know how to deal with this situation?

Thank you.

Answer 1

The issue that you are running into is because you are using the fit_transform method on both your training and test dataset. The correct way of using a transformer is to use fit_transform only on the training dataset so that it learns the parameters and applies the transformation, and then use transform on your test set to apply those learned parameters to transform your data.

Answer 2

Here is a very simple way to do label encoding.

import pandas as pd
 
# Intitialise data of lists
data = [{'Year': 2020, 'Airport':2000, 'Casino':5000, 'Stadium':9000, 'Size':'Small'}, 
       {'Year': 2019, 'Airport':3000, 'Casino':4000, 'Stadium':12000, 'Size':'Medium'},
       {'Year': 2018, 'Airport':5000, 'Casino':9000, 'Stadium':10000, 'Size':'Medium'},
       {'Year': 2017, 'Airport':5000, 'Casino':10000, 'Stadium':15000, 'Size':'Large'}]
df = pd.DataFrame(data)


df = df.set_index(['Year'])
df

df_fin = pd.DataFrame({col: df[col].astype('category').cat.codes for col in df}, index=df.index)
df_fin

Now, you can set your X & Y, get training samples and do testing, and basically do whatever you need to do!