I have a sorted sequence of integers, e.g.
representing volumes in some categories. The task is to separate the valid data (high volumes) from the outliers (low volumes).
In the above example I expect following obvious split
1,2 | 480,1000,1100
This is of course trivial, if there is a definition of outliers, such as less than .1% of the total volume, but I'd prefer to have an algorithm for calculating this threshold as a result.
I don't know anything about the distribution of the values, except for the outliers have low values and the regular categories have high volumes.
Intuitively, I'd say the approach should pass all possible N-1 splits of the sequence and evaluate some criterion, but I fail to find a meaningful one (e.g. I distinctly failed comparing the WSS of the sequence with the sum of the WSS of both splits).
I'd appreciate hints to the algorithm or to transformation of this problem in other well known unsupervised problem.