Removing Outliers within R Works on One Attribute But Not on Second

Question

(Note: this is part of a homework assignment. I am not asking for a solution to the assigned problem, I am just confused by some behavior in R.)

I have a dataset loaded into an R script as a dataframe. After displaying three of the attributes in a boxplot, I have identified the presence of outliers in two of the three. The next step of the assignment is to remove all outliers. I have written code to do this, but it only removes them from the first of the two attributes; the second application of the process doesn't remove any additional records from the dataframe.

I am using the following code, adapted from reading this R-bloggers post:

# Get quantiles and IQR for each of the two attributes:
ar_quantiles <- quantile(leaf_data$aspect_ratio)
ar_iqr <- IQR(leaf_data$aspect_ratio)
s_quantiles <- quantile(leaf_data$solidity)
s_iqr <- IQR(leaf_data$solidity)

# Aspect Ratio cleanup.
leaf_data <- subset(
    leaf_data,
    leaf_data$aspect_ratio > (ar_quantiles[1] - 1.5 * ar_iqr) &
leaf_data$aspect_ratio < (ar_quantiles[2] + 1.5 * ar_iqr)
)

# Solidity cleanup.
leaf_data <- subset(
    leaf_data,
    leaf_data$solidity > (s_quantiles[1] - 1.5 * s_iqr) &
leaf_data$solidity < (s_quantiles[2] + 1.5 * s_iqr)
)

(There was a different method demonstrated in the blog post, but when I switched to that method it left an outlier in the aspect_ratio column and still made no change to the solidity column.)

Because of my limited R understanding, the only real debugging I've been able to do is to dump the output of boxplot(leaf_data$solidity, plot = FALSE)$out before and after the lines that clean up the solidity column. But there was no change in the number of the elements returned by that expression (the elements themselves were unchanged as well).

Does this look like a common error/misunderstanding to anyone?

Answer 1

The default quantiles with quantile are 0%, 25%, 50%, 75%, 100%.

This means that the quartiles Q1 and Q3 are the second and fourth value in the two vector you obtain.

Since indexes start at 1 it means that you can obtain Q1 with ar_quantiles[2] and Q3 with ar_quantiles[4]. Currently you're using indexes 1 and 2 and that's certainly what causes the problem.

[edit] Another option is to specify only the quantiles which are needed when calling quantile:

ar_quantiles <- quantile(leaf_data$aspect_ratio,probs=c(.25,.75))

This way the resulting vector contains only positions 1 and 2 and the rest of your code would work.

Note that you can print your quantiles vector like this:

print(ar_quantiles)

And you can access the documentation for any function like this:

?quantile