2
$\begingroup$

I am experimenting with a dataset and I have a couple of columns with high cardinality. So, I performed mean target encoding (given that my dataset had more than 50000 observations). But, before doing this, I split my data (using Pandas) into training and test and then encoded them. After checking, I realized that my training and testing datasets had Nan's. Since you can not run an ML algorithm with Nan's, I decided to impute both of them with MissForest.

My question is: Is what I have done acceptable?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Data Encoding+imputation(preparation) should be done separately for training and test data, otherwise test data may influence your training and hence can generate a biased model. $\endgroup$
    – vipin bansal
    Jun 20, 2019 at 5:37

2 Answers 2

Reset to default
1
$\begingroup$

Yes, it is possible to impute both the train and the test set. You have to be careful not to introduce information leakage by splitting - if you impute for the train set, then use the same imputation process for the test set as well. I believe that was mentioned in a comment as well. Here is some further information:

See https://stats.stackexchange.com/questions/95083/imputation-before-or-after-splitting-into-train-and-test

Improve this answer
$\endgroup$
0
$\begingroup$

You should not impute your testing set unless you know you can get that data in real life. Most of the time imputing just makes zero sense in real life data. Let's say for example you're predicting whether a customer will default and take loan application variables, if someone forgot to fill his age in application form, you will not use your preset training imputation model to fill this up, it is not just a data point missing by random, it's a real person. So be careful about what you're using the model for and then decide on imputation of test.

Improve this answer
$\endgroup$
6
  • $\begingroup$ That's what I thought too , thank you for the information and the confirmation. $\endgroup$
    – Dimi
    May 20, 2019 at 20:20
  • 1
    $\begingroup$ I am not too sure I follow what this answer is trying to convey. If some new data is missing and the data is not missing at random, why exactly can I not make a best guess on what the missing field is, given what I have seen in the training set/past and based on other variables that aren't missing? In your example, are you trying to state that the person's age would be obtained if it were missing? That really has nothing to do with imputation then. $\endgroup$
    – aranglol
    May 20, 2019 at 22:21
  • $\begingroup$ You cannot make "best guess" for some variables in the same way you cannot predict any variable. Simple example : If you're doing a linear regression that has an independent variable as age of senior manager of a company and the dependent variable as sales for the company, would it make sense to do regression the other way around? Predict age with sales? No, because there has to be an economic significance with anything you're trying to predict. Same case is with most of the impute variables, it makes zero economic sense to make a guess about that, because by nature they are not predictable $\endgroup$
    – Dhruv Mahajan
    May 21, 2019 at 2:51
  • $\begingroup$ I am not talking about using the dependant variable but other independant variables in the analysis. What is stopping me from using many other different independant variables to impute age, for example? I could use Position, years of employment, and other independant variables to impute age. This is exactly how many internal and external imputation methods work, like kNN, gradient boosting, MICE, the list goes on. Of course they are different in how they do it but they still revolve around using other independant variables to impute missing values... $\endgroup$
    – aranglol
    May 21, 2019 at 19:00
  • 1
    $\begingroup$ Yeah, I don't see why imputation is "disallowed" in real life. On average, surely your prediction model can do better if it uses a "most likely" value rather than nothing at all. For your missing age example, you could compute a risk score based on the average age of loan applicants, and that will do better than knowing absolutely nothing about age, which would weight the risk score of a 20-, 40-, or 90-year-old person equally. If 75% of your applicants are 30, there's no sense in treating a missing age value as completely uninformative - they're probably 30. $\endgroup$
    – Nuclear Hoagie
    Oct 18, 2019 at 18:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.