0
$\begingroup$

I am trying to perform OneHotEncoding as well as feature scaling on my training and testing data separately, steps I did:

X = df.drop("target", axis = 1)
y = df.target

X_train,X_test,y_train,y_test = train_test_split(X, y, test_size=0.2)

categorial_features = [List of Categorical Features]
numerical_features = [List of Numerical Features]

one_hot = OneHotEncoder()
scaler = StandardScaler()
tranformer = ColumnTransformer([("one_hot", one_hot, categorial_features),("standard_scaler", scaler, numerical_features)], remainder = "passthrough")

transformed_X_train = tranformer.fit_transform(X_train)
transformed_X_test = tranformer.fit_transform(X_test)

But now the shapes of transformed_X_train, and transformed_X_test are different, I know the reason why it is different, but I want to know how to deal with this situation?

Thank you.

$\endgroup$
3
$\begingroup$

The issue that you are running into is because you are using the fit_transform method on both your training and test dataset. The correct way of using a transformer is to use fit_transform only on the training dataset so that it learns the parameters and applies the transformation, and then use transform on your test set to apply those learned parameters to transform your data.

$\endgroup$
4
  • $\begingroup$ Hi can you show me the code example to do the same for the scenario I have mentioned in the question, that would be really really helpful, I am new to ML. Thank you. $\endgroup$ Sep 28 at 14:03
  • 1
    $\begingroup$ Not sure what code example you need, but it is as simply as changing the second fit_transform call to transform to you only transform your test dataset. $\endgroup$
    – Oxbowerce
    Sep 28 at 15:04
  • $\begingroup$ One changing fit_transform to transform it is still causing the same shape disparity error. $\endgroup$ Sep 28 at 15:57
  • $\begingroup$ Which one are you changing? You have to change the second one from fit_transform to transform. $\endgroup$
    – Oxbowerce
    Sep 28 at 16:11
1
$\begingroup$

Here is a very simple way to do label encoding.

import pandas as pd
 
# Intitialise data of lists
data = [{'Year': 2020, 'Airport':2000, 'Casino':5000, 'Stadium':9000, 'Size':'Small'}, 
       {'Year': 2019, 'Airport':3000, 'Casino':4000, 'Stadium':12000, 'Size':'Medium'},
       {'Year': 2018, 'Airport':5000, 'Casino':9000, 'Stadium':10000, 'Size':'Medium'},
       {'Year': 2017, 'Airport':5000, 'Casino':10000, 'Stadium':15000, 'Size':'Large'}]
df = pd.DataFrame(data)


df = df.set_index(['Year'])
df

df_fin = pd.DataFrame({col: df[col].astype('category').cat.codes for col in df}, index=df.index)
df_fin

enter image description here

Now, you can set your X & Y, get training samples and do testing, and basically do whatever you need to do!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.