# Outlier Elimination in Spark With InterQuartileRange Results in Error

I have the following function that is supposed to calculate the outlier for a given dataset.

def interQuartileRangeFiltering(df: DataFrame): DataFrame = {
@scala.annotation.tailrec
def inner(cols: List[String], acc: DataFrame): DataFrame = cols match {
case Nil          => acc
case column :: xs =>
val quantiles = acc.stat.approxQuantile(column, Array(0.25, 0.75), 0.0) // TODO: values should come from config
println(s"$$column$${quantiles.size}")
val q1 = quantiles(0)
val q3 = quantiles(1)
val iqr = q1 - q3
val lowerRange = q1 - 1.5 * iqr
val upperRange = q3 + 1.5 * iqr
val filtered = acc.filter(s"$$column <$$lowerRange or $$column >$$upperRange")
inner(xs, filtered)
}
inner(df.columns.toList, df)
}

val outlierDF = interQuartileRangeFiltering(incomingDF)


But what happens is that, I have a few features in the incomingDF that are categorical, or in other words binary types with a value of 0 or 1. If I include them, I end up getting an error as below:

housing_median_age 2
inland 2
island 2
population 2
total_bedrooms 2
near_bay 2
near_ocean 2
median_house_value 0
java.lang.ArrayIndexOutOfBoundsException: 0
at inner\$1(<console>:75)
at interQuartileRangeFiltering(<console>:83)
... 54 elided


I have a few questions on how to deal with Outliers for data that is either a 0 or a 1. I can ignore them when doing IQR and this seems to be a reasonable approach, but now my question is, if I ignore them, then how will I join the resulting DataFrame (after running through the recursive function above) back with the OneHotEncoded columns?

For example., if the original dataframe, in this case the incomingDF contains 10000 rows and after outlier detection, it ends up being around 9000 rows, then the excluded columns (the OneHotEncoded columns) still have 10000 and how am I going to merge these two dataframes? Somehow this is confusing to me.

Could someone please help me a way out?

## 1 Answer

You have at least two options:

• Option 1: pass the full dataframe to the functions, possibly with an argument specifying which columns to proceed or to ignore. Whenever an outlier is found, the full row is removed immediately and at the end the resulting dataframe is returned with all its columns.
• Option 2: the function works only on the numerical columns, but instead of removing the rows directly it returns a list of row indexes to remove. Then the rows can be returned from any dataframe which had the same original number of rows.
• Could you please elaborate on your statements? I'm indeed passing the full DataFrame to the function and I can't quite make out what you are trying to say with both the options. Jan 6 at 1:06
• @joesan let me explain the second idea, it's probably closer to what you're doing: currently the function calculates the outliers for every column and proceeds immediately to filter them out. Instead the function would do only the calculation and (instead of applying the filtering) it would return the row numbers of all the outliers found. Later this list of row numbers can be applied to remove the outliers on this dataframe and also on the other dataframe containing the OHE columns. It's just a matter of keeping the correspondence between the rows in the two dataframes. Jan 6 at 19:36