Finding outlier for millions of data that can be used cluster-based algorithm

Question

I have millions of data that each have many features. But as long as the value of a feature is not in the acceptable range, that data will be considered an outlier. And I need to find the acceptable range for each feature.

For example: a data have 3 features A, B, C:

• The smaller the value of A, the better it is considered. So if the value of A is bigger than 20 then data is outlier (given most data has A < 20).
• The bigger the value of B, the better it is considered. So if the value of B is lower than 0.8 then data is outlier (given most data has B > 0.8).
• Most values of C is in range (100, 1000) but some values of C can really big like 5000 or 35000.

My question is:

1. Should I consider data as univariate or multivariable?

2. If I treat my data as univariate, will Local outliers or other advanced methods like Isolated Forest have an advantage over IQR?

3. I have a simple picture of the data distribution as below (just an assumption). With blue and grean dots is good data and red dots is outlier.

• If I use LOF or Isolated Forest (IF), then green dots may be consider as outlier.
• So I am thinking about using LOF (or IF,...) to find blue dots cluster. Then, based on this blue dots cluster, I will find the maximum value of feature A (Max_A) and the minimum value of feature B (Min_B) in this cluster.
• Then the acceptable range will be (0, Max_A) for feature A and (Min_B, Inf) for feature B.
• But I'm not sure if this is the right way to do it. Is this normal to use LOF,... like this way?

Thank you for reading!

Answer 1

1. Should I consider data as univariate or multivariable?

This depends on the nature of your data and the relationships between the features. If the features are independent of each other, you can treat them as univariate. However, if there are correlations or dependencies between the features, it would be more appropriate to treat them as multivariate.

2. If I treat my data as univariate, will Local outliers or other advanced methods like Isolated Forest have an advantage over IQR?

Again, this depends on the nature of your data. The Interquartile Range (IQR) method is a simple and effective way to detect outliers in univariate data, but it may not be as effective for multivariate data or data with complex distributions. Local Outlier Factor (LOF) and Isolation Forest are more advanced methods that can handle multivariate data and complex distributions better than IQR.

3. Is this normal to use LOF,... like this way?

Your approach of using LOF or Isolation Forest to find the main cluster of data and then defining the acceptable range based on this cluster seems reasonable. However, keep in mind that these methods are not perfect and may not always give the desired results. It's always a good idea to validate your results with other methods or with expert knowledge if available.

In addition, you might want to consider using a robust method to estimate the maximum and minimum values in your acceptable range. For example, instead of simply taking the maximum and minimum values in your main cluster, you could use a method that is less sensitive to outliers, such as the median or a trimmed mean.

In conclusion, the best approach depends on the specific characteristics of your data. It may be helpful to try several different methods and compare their performance.