# Gracefully removing observations with outliers in N fields

I have a function.

remove_outliers <- function(x, na.rm = TRUE, ...) {

#find position of 1st and 3rd quantile not including NA's
qnt <- quantile(x, probs=c(.25, .75), na.rm = na.rm, ...)

H <- 1.5 * IQR(x, na.rm = na.rm)

y <- x
y[x < (qnt - H)] <- NA
y[x > (qnt + H)] <- NA
x<-y

#get rid of any NA's
x[!is.na(x)]
}


Given a dataset(numbers) like this:

  x
5
9
2
99
3
4


The functioning is obvious

remove_outliers(numbers)


means I now have this:

  x
5
9
2
3
4


However, what if I have an ID that I want to retain, such as:

number_id    numbers
12              5
23              9
34              2
45              99
56              3
67              4


How do I remove the outlier(99) with the remove_outliers function(or another, better suited function), to get this data:

number_id    numbers
12              5
23              9
34              2
56              3
67              4


(note the entire observation with the outlier has been removed)

And how can I scale this solution to handle n more variables?

I can do it very ungracefully by taking out each column separately and building a new data frame with loops, but it's hardly readable and a mess to debug. Is there a more graceful way?

This will achieve what you want. You can remove outliers from any column you wish, just pass that column number as an argument in the function.

id <- c(12,23,34,45,56,67)
num <- c(5,9,2,99,3,4)
prac <- data.frame(id, num)

remove_outliers <- function(x, col) {

#find position of 1st and 3rd quantile not including NA's
qnt <- quantile(x[ ,col], probs=c(.25, .75), na.rm = TRUE)

H <- 1.5 * IQR(x[ ,col])

x[ ,col] <- ifelse(x[ ,col] < (qnt - H) | x[ ,col] > (qnt + H), NA, x[ ,col])

#get rid of any NA's
x <- x[!is.na(x[ ,col]), ]
x <- assign("dataset", x, envir = .GlobalEnv)
return(x)
}

remove_outliers(prac, 2)

• After adding the second parameter when the function is called(remove_outliers(prac, "col_name") it works as expected until I get: "Error in quantile.default(as.numeric(x), c(0.25, 0.75), na.rm = na.rm, : missing values and NaN's not allowed if 'na.rm' is FALSE" But it's not false, and the line na.rm = na.rm isn't even part of this function anymore. All variables are ints. – aabreu Feb 21 '19 at 10:27
• If you're referencing columns by name and $, I'm not sure why, but I've had a lot of trouble with this in functions. That is why I use [ ,col]. Just enter the column number that you want to remove. I will edit the post as I realized it isn't there. I do not receive that error and this method works for any data set I've used so long as you reference column number not name – mmb_rach Feb 21 '19 at 22:10 • I seems to work decently by name over here. Not with "$", just "col_name", at least there is no difference in my output if I reference by name or number. It's a big dataset(30+ variables and 300K observations) so referencing by number is very unreadable(IMO). – aabreu Feb 22 '19 at 10:02