1
$\begingroup$

I have a dataset with 60 columns and 5K records. There are few columns which has outliers. I understand that there are multiple approach to handle outliers.

Actually I don't wish to drop the data as it is an outlier because am not sure whether it is really an outlier or a meaningful value (like how income could be skewed).

So what I did is just standardize the columns using standard scaler.

Is it okay to do this? Am assuming that by doing this standard scaling, I have retained the data as well as got rid of outliers.

I did refer couple of posts (post1, post2) but couldn't get an idea

Is this what standardization does? Can anyone suggest me is there anyother way to handle outliers without dropping the records?

Can you help?

| improve this question | | | | |
$\endgroup$
1
$\begingroup$

Standardising may not be the best option.

Because they will still not be bounded (like when normalised) between -1 and 1 but be distribution dependent. What I mean is if they are outliers their standard deviation will be big for these values. In any case its not that you should rescale the values to combat these outliers. Outliers are not in the first place problematic because they are big in values, but mostly because these values lie in the cluster thats not possible/representative of future data and you will learn on it.

One solution for without dropping them is for example binning, where you say all outliers that have standard deviation bigger than two should have value "X" where X is some 99,5% quantile for example.

| improve this answer | | | | |
$\endgroup$
  • $\begingroup$ Thanks upvoted for your help. I would like to understand a bit about your suggestion on binning. Can you provide an example to help me understand. Let's say that I have values like 200,300,350,245,560,10000. Here 10000 is an outlier $\endgroup$ – The Great Dec 26 '19 at 10:17
  • $\begingroup$ Whats standard devitation of your baby set (with or without 10 000) its 500 for example, 2 times this is 1000 s you would set all values that are bigger than 1000 to 1000 $\endgroup$ – Noah Weber Dec 26 '19 at 10:19
  • $\begingroup$ got you. Anything above 2sd, should be set to the maximum value (within 2sd). got it. marked as answer $\endgroup$ – The Great Dec 26 '19 at 10:20
1
$\begingroup$

When you dont want to remove outlier then you can either use logistic regression if it is classification task as we know log reg is robust to outliers because of sigmoidal function. Otherwise for general you can use local reachability factor value to check whether you datapoint is having this factor value significantly different from other datapoints. Local reachability factor and remove these higher value datapoints from the dataset.

| improve this answer | | | | |
$\endgroup$
  • $\begingroup$ thanks for the response. Upvoted. So, standardizing may not be the best option? $\endgroup$ – The Great Dec 26 '19 at 6:05
0
$\begingroup$

No, simply if you standardize you are taking outlier into consideration and if you are using sensitive model which gets affected with outlier like KNN etc then it is not good practice to do that. I suggest either choose distance unsensitive model or remove outliers then only your performance may increase.

| improve this answer | | | | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.