Suppose I have a train and test data set and both of them contain missing values. Can I join both of them and then do fillna.(alldata.mean()) or will that cause data leakage. What would be a better way because doing them separately would have different mean for same column.
$\begingroup$
$\endgroup$
asked Mar 22, 2018 at 15:55
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
This is a controversial subject with no clear best answer and myriad options, some of which are model-specific. You can drop them, replace them with extreme values, interpolate, replace with median, impute with nearest neighbors, etc. In all cases besides dropping them altogether, you make assumptions and create data where no data exists. I can't tell you the best approach, but I can advise you to try multiple approaches and hope the end results are robust to choice.
$\begingroup$
$\endgroup$
No you shouldn't do that as the statistics of your training set and your test set isn't the same..
We apply the same transformations on both the data-sets but not using the same statistics..
Also why always filling by mean is preferred?
Search for interpolation in data-frames (requires scipy)
