0
$\begingroup$

I have a sorted sequence of integers, e.g.

1,2,480,1000,1100

representing volumes in some categories. The task is to separate the valid data (high volumes) from the outliers (low volumes).

In the above example I expect following obvious split

1,2   |   480,1000,1100

This is of course trivial, if there is a definition of outliers, such as less than .1% of the total volume, but I'd prefer to have an algorithm for calculating this threshold as a result.

I don't know anything about the distribution of the values, except for the outliers have low values and the regular categories have high volumes.

Intuitively, I'd say the approach should pass all possible N-1 splits of the sequence and evaluate some criterion, but I fail to find a meaningful one (e.g. I distinctly failed comparing the WSS of the sequence with the sum of the WSS of both splits).

I'd appreciate hints to the algorithm or to transformation of this problem in other well known unsupervised problem.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

If the high volume and low volume data both appear in non-negligible probability, you may view the problem as a clustering problem with two clusters. Gaussian mixture model or K-means algorithm may help you in this case.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for pointing to mixture models - I thing this could be relevant. Clustering on other site, seems not helpful for my problem, especially K-Means, as the K (the number of the main categories) is the sought solution. $\endgroup$
    – Marmite Bomber
    Commented Oct 28, 2015 at 8:27
  • $\begingroup$ Note that 1) the normal data itself may have more than one distribution 2) the outlier "low volume" class may occur very infrequently compared to the normal data ... as is usually the case in outlier analysis. These would be important considerations in the suggested approach. $\endgroup$
    – Aman
    Commented Oct 29, 2015 at 16:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.