0
$\begingroup$

I have a sorted sequence of integers, e.g.

1,2,480,1000,1100

representing volumes in some categories. The task is to separate the valid data (high volumes) from the outliers (low volumes).

In the above example I expect following obvious split

1,2   |   480,1000,1100

This is of course trivial, if there is a definition of outliers, such as less than .1% of the total volume, but I'd prefer to have an algorithm for calculating this threshold as a result.

I don't know anything about the distribution of the values, except for the outliers have low values and the regular categories have high volumes.

Intuitively, I'd say the approach should pass all possible N-1 splits of the sequence and evaluate some criterion, but I fail to find a meaningful one (e.g. I distinctly failed comparing the WSS of the sequence with the sum of the WSS of both splits).

I'd appreciate hints to the algorithm or to transformation of this problem in other well known unsupervised problem.

$\endgroup$
1
$\begingroup$

If the high volume and low volume data both appear in non-negligible probability, you may view the problem as a clustering problem with two clusters. Gaussian mixture model or K-means algorithm may help you in this case.

$\endgroup$
  • $\begingroup$ Thanks for pointing to mixture models - I thing this could be relevant. Clustering on other site, seems not helpful for my problem, especially K-Means, as the K (the number of the main categories) is the sought solution. $\endgroup$ – Marmite Bomber Oct 28 '15 at 8:27
  • $\begingroup$ Note that 1) the normal data itself may have more than one distribution 2) the outlier "low volume" class may occur very infrequently compared to the normal data ... as is usually the case in outlier analysis. These would be important considerations in the suggested approach. $\endgroup$ – Aman Oct 29 '15 at 16:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.