0
$\begingroup$

I have two files, a training data with a label field and a test data without the label field. I have plotted a field "A" in train data:

enter image description here

It looks like outliers are 4,5,6 and should be removed.

Now plotting the field "A" in test file also shows somewhat similar results to the image shown. In that case, should I consider 4,5,6 as outliers, and remove them from train data, or keep them? Any pointers are appreciated.

$\endgroup$
1
$\begingroup$

In general, its not acceptable to drop an observation just because it is an outlier.

Sometimes outliers give us very important information about data. Removing outliers is legitimate if:

  • it is obvious that's incorrect data
  • not change the results and assumptions.

How to handle outliers? Try other models or use data transformation.

$\endgroup$
6
  • $\begingroup$ Thanks, could you elaborate on "other models" or "data transformation"? May be you could provide a link to some tutorials? I am not familiar with any of them $\endgroup$ – Afia R. S. Apr 25 '20 at 18:15
  • $\begingroup$ About "other models"- it depends on your data and assumptions, I can't answer without more details. "data transformation"- try for example log transformation, sample have you in this link: chrisalbon.com/machine_learning/preprocessing_structured_data/… $\endgroup$ – fuwiak Apr 25 '20 at 18:26
  • $\begingroup$ I do not agree. It could be ok depending on the case. If you are interested in predicting a limited range in the case of a regression, you will achieve best results by dropping outliers (observations out of the range). $\endgroup$ – Rusoiba Apr 25 '20 at 18:26
  • $\begingroup$ Rusoiba, outliers sometimes they are the most important data of all, how do you justify removing most valuable data from the dataset? $\endgroup$ – fuwiak Apr 25 '20 at 18:34
  • $\begingroup$ If I drop them from my training data, then will similar points in the test files be correctly labeled? Is there a general best practice, when one has such data in both test and train files? $\endgroup$ – Afia R. S. Apr 25 '20 at 18:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.