Predict data using Pre-Trained Classification Model

Question

I have pre trained classification model (saved as pickle file) to predict employee attrition.

My question is when I use new dataset to predict using Pickle file do I need do all preprocessing steps (like transformation and encoding) to the new testing dataset or can I use raw data set?

Answer 1

Yes. Whatever steps/processing you have done to the data before feeding it to the model, all of steps needs to be done again in the raw data. Ideally you should create a function which takes the data/dataframe as input and processes the data. The function should include all the exact steps you did while preprocessing the data when training your model. If you have used feature engineer techniques be sure to include them too, either in the same function or a different one. Here is an example for better understanding. The steps you see below are the steps which I used to preprocess my training data before feeding it to the model. Hope it helps.

def clean_data(df):
df = df.drop_duplicates()
df = df[df.department != 'temp']
df['filed_complaint'] = df.filed_complaint.fillna(0)
df['recently_promoted'] = df.recently_promoted.fillna(0)
df.department.replace('information_technology', 'IT', inplace=True)
df['department'].fillna('Missing', inplace=True)
df['last_evaluation_missing'] = df.last_evaluation.isnull().astype(int)
df.last_evaluation.fillna(0, inplace=True)
return df