I have millions of data that each have many features. But as long as the value of a feature is not in the acceptable range, that data will be considered an outlier. And I need to find the acceptable range for each feature.
For example: a data have 3 features A, B, C:
- The smaller the value of A, the better it is considered. So if the value of A is bigger than 20 then data is outlier (given most data has A < 20).
- The bigger the value of B, the better it is considered. So if the value of B is lower than 0.8 then data is outlier (given most data has B > 0.8).
- Most values of C is in range (100, 1000) but some values of C can really big like 5000 or 35000.
My question is:
Should I consider data as univariate or multivariable?
If I treat my data as univariate, will Local outliers or other advanced methods like Isolated Forest have an advantage over IQR?
I have a simple picture of the data distribution as below (just an assumption). With blue and grean dots is good data and red dots is outlier.
- If I use LOF or Isolated Forest (IF), then green dots may be consider as outlier.
- So I am thinking about using LOF (or IF,...) to find blue dots cluster. Then, based on this blue dots cluster, I will find the maximum value of feature A (Max_A) and the minimum value of feature B (Min_B) in this cluster.
- Then the acceptable range will be (0, Max_A) for feature A and (Min_B, Inf) for feature B.
- But I'm not sure if this is the right way to do it. Is this normal to use LOF,... like this way?
Thank you for reading!