Dealing with outliers

Question

I'm doing some data analysis on the UCI "Adult Dataset". I have a numerical feature called "hours-per-week" and another feature called "age". These are the only numerical features I'm considering in the dataset. I did a boxplot for each of the feature to identify the presence of outliers, like this.

# Select the numerical variables of interest
num_vars = ['age', 'hours-per-week']

# Create a dataframe with the numerical variables
data = df[num_vars]

# Plot side by side vertical boxplots for each variable
fig, axes = plt.subplots(nrows=1, ncols=len(num_vars), figsize=(10,5))
for i, var in enumerate(num_vars):
   sns.boxplot(y=var, data=data, ax=axes[i])
   axes[i].set_ylabel(var)
plt.tight_layout()
plt.show() 

Here's the output:

By winsorizing the data, I was able to take care of the "age" outliers. However, it did not work so well for the "hours-per-week" outliers:

from scipy.stats.mstats import winsorize

# Winsorize the 'age' column
age_wins = winsorize(df['age'], limits=[0.05, 0.05])

# Winsorize the 'hours-per-week' column
hours_wins = winsorize(df['hours-per-week'], limits=[0.05, 0.1])

# Create a new dataframe without the outliers
df_wins = df.assign(age=age_wins, hours_per_week=hours_wins)



# Select the numerical variables of interest
num_vars = ['age', 'hours-per-week']

# Create a dataframe with the numerical variables
data = df_wins[num_vars]

# Plot side by side vertical boxplots for each variable
fig, axes = plt.subplots(nrows=1, ncols=len(num_vars), figsize=(10,5))
for i, var in enumerate(num_vars):
    sns.boxplot(y=var, data=data, ax=axes[i])
    axes[i].set_ylabel(var)
plt.tight_layout()
plt.show()

Here's the output after this:

I tried to increase the limits when winsorizing the data for the "hours-per-week" feature, but it also did not work and doesn't seem to be the best way for dealing with it.

Here's the distribution of the variable in question:

I don't know if I should remove this outliers or not, my goal is to later implement this data in classification machine learning models.

I tried removing the outliers using the IQR method besides winsorizing the data, but it also didn't remove those outliers and also doesn't seem to be the correct way of dealing with this.

Answer 1

Let's take a step back: why do we remove outliers?

Answer: because they hinder our application. That being said, in some applications outliers are so useful that we do not want to throw away.

For example, say I have a feature of how frequent (times per week) customers buy from my store, and there are a few outliers with very high purchase frequency. If my aim is to 'classify customers into high and low lifetime values', shall I throw these 'outliers' away? Probably not. They are valuable customers.

Another example. Say I have a few temperature sensors out in the wild. Usually they read between 10-30 degree Celsius, but in one occasion one reads 200.

Should I remove this 'outlier'? Probably yes, as it was a sensor malfunction, most likely. However, I would definitely not do this if those sensors are positioned inside a nuclear plant, for which I am writing an alert system.

Back to your question: how you deal with outliers depends on your exact application; since you only generically state the application is 'classification machine learning models', I would suggest you to keep all data points, until you have a solid application. Chances are those 'outliers' turn out to be valuable for you work.