0
$\begingroup$

I have two files, a training data with a label field and a test data without the label field. I have plotted a field "A" in train data:

enter image description here

It looks like outliers are 4,5,6 and should be removed.

Now plotting the field "A" in test file also shows somewhat similar results to the image shown. In that case, should I consider 4,5,6 as outliers, and remove them from train data, or keep them? Any pointers are appreciated.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

In general, its not acceptable to drop an observation just because it is an outlier.

Sometimes outliers give us very important information about data. Removing outliers is legitimate if:

  • it is obvious that's incorrect data
  • not change the results and assumptions.

How to handle outliers? Try other models or use data transformation.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks, could you elaborate on "other models" or "data transformation"? May be you could provide a link to some tutorials? I am not familiar with any of them $\endgroup$
    – Afia R. S.
    Commented Apr 25, 2020 at 18:15
  • $\begingroup$ About "other models"- it depends on your data and assumptions, I can't answer without more details. "data transformation"- try for example log transformation, sample have you in this link: chrisalbon.com/machine_learning/preprocessing_structured_data/… $\endgroup$
    – fuwiak
    Commented Apr 25, 2020 at 18:26
  • $\begingroup$ I do not agree. It could be ok depending on the case. If you are interested in predicting a limited range in the case of a regression, you will achieve best results by dropping outliers (observations out of the range). $\endgroup$
    – Rusoiba
    Commented Apr 25, 2020 at 18:26
  • $\begingroup$ Rusoiba, outliers sometimes they are the most important data of all, how do you justify removing most valuable data from the dataset? $\endgroup$
    – fuwiak
    Commented Apr 25, 2020 at 18:34
  • $\begingroup$ If I drop them from my training data, then will similar points in the test files be correctly labeled? Is there a general best practice, when one has such data in both test and train files? $\endgroup$
    – Afia R. S.
    Commented Apr 25, 2020 at 18:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.