2
$\begingroup$

I have a function.

remove_outliers <- function(x, na.rm = TRUE, ...) {

    #find position of 1st and 3rd quantile not including NA's
    qnt <- quantile(x, probs=c(.25, .75), na.rm = na.rm, ...)

    H <- 1.5 * IQR(x, na.rm = na.rm)

    y <- x
    y[x < (qnt[1] - H)] <- NA
    y[x > (qnt[2] + H)] <- NA
    x<-y

    #get rid of any NA's
    x[!is.na(x)]
}

Given a dataset(numbers) like this:

  x
  5
  9
  2
  99
  3
  4

The functioning is obvious

remove_outliers(numbers)

means I now have this:

  x
  5
  9
  2
  3
  4

However, what if I have an ID that I want to retain, such as:

number_id    numbers
12              5
23              9
34              2
45              99
56              3
67              4

How do I remove the outlier(99) with the remove_outliers function(or another, better suited function), to get this data:

number_id    numbers
12              5
23              9
34              2
56              3
67              4

(note the entire observation with the outlier has been removed)

And how can I scale this solution to handle n more variables?

I can do it very ungracefully by taking out each column separately and building a new data frame with loops, but it's hardly readable and a mess to debug. Is there a more graceful way?

$\endgroup$
1
$\begingroup$

This will achieve what you want. You can remove outliers from any column you wish, just pass that column number as an argument in the function.

id <- c(12,23,34,45,56,67)
num <- c(5,9,2,99,3,4)
prac <- data.frame(id, num)


remove_outliers <- function(x, col) {

  #find position of 1st and 3rd quantile not including NA's
  qnt <- quantile(x[ ,col], probs=c(.25, .75), na.rm = TRUE)

  H <- 1.5 * IQR(x[ ,col])

  x[ ,col] <- ifelse(x[ ,col] < (qnt[1] - H) | x[ ,col] > (qnt[2] + H), NA, x[ ,col])


  #get rid of any NA's
  x <- x[!is.na(x[ ,col]), ]
  x <- assign("dataset", x, envir = .GlobalEnv)
  return(x)
}

remove_outliers(prac, 2)
$\endgroup$
  • $\begingroup$ After adding the second parameter when the function is called(remove_outliers(prac, "col_name") it works as expected until I get: "Error in quantile.default(as.numeric(x), c(0.25, 0.75), na.rm = na.rm, : missing values and NaN's not allowed if 'na.rm' is FALSE" But it's not false, and the line na.rm = na.rm isn't even part of this function anymore. All variables are ints. $\endgroup$ – aabreu Feb 21 at 10:27
  • $\begingroup$ If you're referencing columns by name and $, I'm not sure why, but I've had a lot of trouble with this in functions. That is why I use [ ,col]. Just enter the column number that you want to remove. I will edit the post as I realized it isn't there. I do not receive that error and this method works for any data set I've used so long as you reference column number not name $\endgroup$ – mmb_rach Feb 21 at 22:10
  • $\begingroup$ I seems to work decently by name over here. Not with "$", just "col_name", at least there is no difference in my output if I reference by name or number. It's a big dataset(30+ variables and 300K observations) so referencing by number is very unreadable(IMO). $\endgroup$ – aabreu Feb 22 at 10:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.