1
$\begingroup$

I am doing some exploratory data analysis on a custom dataset. I have 3 different data-frames:

  1. df1 belonging to class 0.
  2. df2 belonging to class 1.
  3. df3 belonging to class 2.

I am doing k-means clustering on the combined data-frame (by combining df1, df2, df3). Should I remove outliers before combining the data-frames or after combining the data-frames?

I am removing outliers using IQR technique. I have been doing it before combining the data-frame but I was wondering if it is the right approach.

Improve this question
$\endgroup$
2
  • $\begingroup$ Why remove so-called outliers at all? $\endgroup$
    – Dave
    Commented Jan 10 at 22:01
  • 1
    $\begingroup$ My next step is to see if k-means clustering does the job of clustering those 3 different classes. That is why I wanted to remove the outliers. $\endgroup$
    – Harshvardhan Uppaluru
    Commented Jan 11 at 0:24

1 Answer 1

Reset to default
0
$\begingroup$

It should not create a negative effect since you mentioned you are doing exploratory data analysis. Keeping or removing outliers affect model performance when we perform machine learning, deep learning, or similar advanced modeling technique.

Improve this answer
$\endgroup$
6
  • $\begingroup$ My next step is to see if k-means clustering does the job of clustering those 3 different classes. That is why I wanted to remove the outliers. $\endgroup$
    – Harshvardhan Uppaluru
    Commented Jan 11 at 0:23
  • $\begingroup$ @HarshvardhanUppaluru ok. For exploratory data analysis no need to remove. For k-means, you need to remove (as needed) and possibly normalize the set. Upvote or mark my answer as correct if I answered your original question (whether or not you should remove outliers before exploratory data analysis). $\endgroup$
    – Full Array
    Commented Jan 11 at 1:35
  • $\begingroup$ What do you mean it should not create a negative effect? The presence of "outliers" can greatly affect the results of say clustering creating very different clusters. $\endgroup$
    – user2974951
    Commented Jan 11 at 8:12
  • $\begingroup$ @user2974951 you are misquoting my comment because you left out ..since you mentioned you are doing "exploratory data analysis". as OP indicated in the original question. $\endgroup$
    – Full Array
    Commented Jan 11 at 12:01
  • $\begingroup$ My comment still holds, it makes no difference if you are doing exploratory or confirmatory analysis, your results will be affected by "outliers". $\endgroup$
    – user2974951
    Commented Jan 11 at 12:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.