0
$\begingroup$

I have a large dataset of >100 columns with nearly all types of data. I want to remove outliers from my dataset for which purpose I've decided to use IQR. Problem is even when I apply quantile of 0.25/0.75, I still get significant amount of outliers in columns like ClientTotalIncome, etc. Further by doing that, I eliminate more than 90% data. My code in Python for outliers removal is as follows:

num_train = train.select_dtypes(include=['number'])
cat_train = train.select_dtypes(exclude=['number'])
Q1 = num_train.quantile(0.25)
Q3 = num_train.quantile(0.75)
IQR = Q3 - Q1
idx = ~((num_train < (Q1 - 1.5 * IQR)) | (num_train > (Q3 + 1.5 * 
IQR))).any(axis=1)
train_cleaned = pd.concat([num_train.loc[idx], cat_train.loc[idx]], axis=1)

Any ideas?

$\endgroup$
3
$\begingroup$

It's not always a good idea to remove data from your dataset.

In some circumstances - and income is a good example - your data will be skewed / long-tailed and so will lie outside of the interquartile range. This doesn't imply that there is anything wrong with the data, but rather that there is a disparity between observations.

Nevertheless, if you are set on removing observations perhaps you should consider scaling your features prior to determining which observations are outliers. For example, taking the log of a feature and then applying your outlier removal based on the log(variable).

Don't forget that IQR doesn't carry over well to categorical and ordered features.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.