1
$\begingroup$

I understand that before feature engineering one has to split the dataset into train and test data, so as to avoid bias in the analysis. I also understand that the machine learning model does not understand data apart from numerical data, thus encoding is required, which is a part of feature engineering. My question is, do I encode the test data separately or does the prediction function understand categorical data.

$\endgroup$
2
  • 1
    $\begingroup$ Depends on the platform. R can handle factors easily. What language you are on? $\endgroup$
    – Peter
    Nov 6, 2019 at 22:07
  • $\begingroup$ I am on python. $\endgroup$
    – Fhunmie
    Nov 7, 2019 at 7:34

2 Answers 2

2
$\begingroup$

This depends somewhat on the model and language (implementation).

First please understand that categorical data is not the same as non-numerical data! Many models can handle categorical data (e.g. regression formats) just fine and some can even handle non-numerical data.

Finally and most important for you feature engineering has to be done on the whole data set before the train/test split. All models can only predict on data that has the exact same input formats as the data it has been trained on!

So yes if you one-hot encoded some column it also needs to be one-hot encoded for the prediction.

$\endgroup$
2
  • $\begingroup$ Thank you so much this answers my question. $\endgroup$
    – Fhunmie
    Nov 7, 2019 at 11:35
  • $\begingroup$ @MercyAkinkuotu Keep in mind that this poses some challenges in using the model on actual data. Depending on your code one-hot-encoding on new data may not produce all needed columns for the model because one factor level is missing (e.g. if gender splits to two columns male & female, your code may produce only one column for a new male data point). Therefore I always recommend to save the factor levels involved in creating the original model and applying them to new data points before transforming them as necessary. $\endgroup$
    – Fnguyen
    Nov 7, 2019 at 11:41
0
$\begingroup$

If you want to ensure that the test data has similar classes to the train set, you can use the stratify option in scikitlearn train test split for Python or the stratify option in Caret for R

$\endgroup$
1
  • 1
    $\begingroup$ Thanks,but what i want to know is that i did one hot encoding for the binary variables in my train data ,should i one hot encode that same binary variables in my test data also. $\endgroup$
    – Fhunmie
    Nov 7, 2019 at 8:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.