2
$\begingroup$

I'm very new to this community, so please overlook my noobness.

I have a data set with 2948 instances and I tried to remove outliers using InterquartileRange filter in Weka. The issue is that the number of 'YES' instances in ExtremeValues and Outliers takes up to 2947 and 2946 respectively. In other words, all my data are considered outliers.

What does this say about my data set? Or am I not meant to perform IQR on this data, if so, is there other algorithms to identify outliers other than IQR? And how would one perform regression on such a data set?

Thank you.

$\endgroup$
  • 1
    $\begingroup$ Something is wrong here. IQR is the gap between the 25th and 75th percentile of your data. By definition this must contain 50% of your data. Is there a parameter on the filter that regulates what multiple of IQR you should consider extreme? Typical values would be 2-3 * IQR $\endgroup$ – Ben Allison Mar 13 '15 at 15:34
1
$\begingroup$

The InterQuartileRangeFilter from weka library uses an IQR formula to designate some values as outliers/extreme values. Any value outside this range $[Q_1 - k(Q_3-Q_1), Q_3 + k(Q_3-Q_1)]$ is considered some sort of an outlier, where $k$ is some constant, and $ IQR = Q_3 - Q_1$.

By default weka uses $k=3$ to define something as outlier, and $k=3*2$ to define something as extreme value (extreme outlier).

The formula guarantees that at least 50% value are considered non-outliers. Having a single variable (univariate sample of values), it's practically impossible to reproduce your result.

Note however that this filter can be applied to a data frame. When applied like this, it will consider as an outlier any instance of the data frame which has at least one value of the instance considered as outlier for that variable.

Now, supposing that you have a data frame with 2 variables, totally uncorrelated (independent). Considering again that only 10% of the values from each variable are considered outliers, due to independence, one can expect that $(1-0.9)^2$ values will not be outliers. If you have $p$ variables like that in your data frame, you might expect to have only $(1-0.9)^k$ normal values, and is not very hard to arrive in that situation.

There are two things which you will have to consider. One is to increase the factors for outliers if in general too many values are considered outliers (ideally you would like to take a look at each variable graphically and if possible to get some idea about the distribution beneath). The second one is to check if you have many values which are totally independent. The second hint does not solve your problem but might give you a reason why it happens.

$\endgroup$
  • $\begingroup$ hmmm... i think you are on to something here. i still need to look at it again. Thanks for the hint though. I will mark it as the correct answer (permanently) since I can't vote up yet. But I might comment again to ask for more clarification (if something comes up) $\endgroup$ – TuanDT May 13 '15 at 2:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.