0
$\begingroup$

I have pre trained classification model (saved as pickle file) to predict employee attrition.

My question is when I use new dataset to predict using Pickle file do I need do all preprocessing steps (like transformation and encoding) to the new testing dataset or can I use raw data set?

$\endgroup$
1
  • $\begingroup$ Not sure if I understand well. Do you want to use the pickel file together with raw data? In all the cases you need to have comparable datasets in the same format to make good training. $\endgroup$ Jun 22 at 7:03

1 Answer 1

1
$\begingroup$

Yes. Whatever steps/processing you have done to the data before feeding it to the model, all of steps needs to be done again in the raw data. Ideally you should create a function which takes the data/dataframe as input and processes the data. The function should include all the exact steps you did while preprocessing the data when training your model. If you have used feature engineer techniques be sure to include them too, either in the same function or a different one. Here is an example for better understanding. The steps you see below are the steps which I used to preprocess my training data before feeding it to the model. Hope it helps.

def clean_data(df):
df = df.drop_duplicates()
df = df[df.department != 'temp']
df['filed_complaint'] = df.filed_complaint.fillna(0)
df['recently_promoted'] = df.recently_promoted.fillna(0)
df.department.replace('information_technology', 'IT', inplace=True)
df['department'].fillna('Missing', inplace=True)
df['last_evaluation_missing'] = df.last_evaluation.isnull().astype(int)
df.last_evaluation.fillna(0, inplace=True)
return df
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.