1
$\begingroup$

I am learning ML. I have a dataframe with some features and a target column. For simplicity condiser X has one feature, for eg X = [1, 2, 3, 4, 5, 6, 6, 1, 2, 12]

One of the pre-processing step is find outliers and impute with mean.

Case-1 Suppose X represent train set, Then it is straight forward find and impute it.

Case-2 Suppose X represent dev or test set, Then what action one should take? A. Should apply the same pre-processing step as we did for train set? B. By pre-processing the outliers on train set, we get a minimum and maximum value/limits for that feature. Now if future data contain value which is out bound with respect to minimum or maximum value of that feature.Should we do predict output for it or we should discard entire record?

Please suggest.

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
3
$\begingroup$

Some pre-processing must be done on the train set and test set, e.g. tokenization for a document, or normalization of variables, but removing outliers should only be done on train set, since you want to know how good your model is on real data (test set) you shouldn't remove outliers for this set.

For the train-dev check 1: you should first remove outliers then make the split. At the end your test error will be (as always should be) greater than your dev error.

This question has been asked before: When should you remove outliers?

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Hi Memristor, Thanks. Make sense, not to remove outliers from test set. $\endgroup$
    – winter
    Commented Jun 13, 2023 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.