3
$\begingroup$

I want to identify outliers from a very small group of numbers. How shall I do that?

For example, for the group of numbers: -0.4, 0.4, 52.1, actually 52.1 is an outlier.

I've tried using interquartile range to identify the outliers, but it won't identify 52.1 as the outlier.

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ You could set a threshold in terms of the number of standard deviations from the mean. $\endgroup$
    – image_doctor
    Commented Aug 26, 2015 at 15:37
  • 1
    $\begingroup$ This stretches the definition of "outlier", maybe to the point that you will get bad results in your work. Is there some knowledge or expectation in general about the numbers you are seeing in your project, that you could somehow include (if so, maybe explain)? Or is there some common theme amongst repeated sets of numbers that could maybe used to learn a rule (as you have tagged this "machine-learning")? $\endgroup$
    – Neil Slater
    Commented Aug 26, 2015 at 19:58

1 Answer 1

Reset to default
5
$\begingroup$

An "outlier" is an observation that is so unexpected that we suspect it wasn't valid -- corrupted by noise or something. But what is unexpected? An observation that is highly improbable? But then how do we know what's probable?

Unless you're able or willing to make some assumptions about the distribution that generated these numbers, you can't really declare things outliers. For example, quartiles don't help, since it will only help you find the x% largest or smallest values. But every data set has these -- indeed every data set has a min and max, and being the min or max doesn't mean being an outlier.

In practice, I sense there's an assumption lurking here, that the numbers are probably normally distributed about some mean with roughly some standard deviation. If you know what the mean and/or stdev is supposed to be, you can use it directly to decide how unlikely an observation is and discard it as an outlier if it exceeds a threshold you choose.

You can take the mean and stdev of this sample as a surrogate for that, and for a large enough sample, the sample mean and stdev could be close enough to the real population mean and stdev to work. Here, with such small sets, the outliers influence stats like the mean so much that they throw off attempts to evaluate them in terms of the stats they influence. It's kind of circular.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thank you for the detailed explanation. To summarise your points, my understanding is as follows. First, we need to know the underlying distribution of a group of numbers to define the outliers. I guess what this implies is data points not lying on the distribution called outliers? Second, mean and stdev can be used to detect outliers only for a large sample. They won't work on small sets of numbers (as for the example that I have given). $\endgroup$
    – Jane Foster
    Commented Aug 28, 2015 at 9:55
  • 1
    $\begingroup$ Kind of.. I'm saying you at least must have some assumption about the distribution; you may not know it for sure. Typically for a continuous distribution like the normal distribution all values are "possible" in the distribution, just some are unlikely. What I mean is that the sample mean/stdev tends to the real population mean/stdev as the sample size increases, which is a little different. For a small sample, it by itself is 'not enough information' even given an assumption of normality. $\endgroup$
    – Sean Owen
    Commented Aug 28, 2015 at 12:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.