I'm trying to determine the best approach to an anomaly detection problem. Particularly around setting up the data, building the models, and leveraging the models to identify important information.
Suppose I want to identify sales anomalies in an e-commerce business with a lot of product diversity. Further, this business has 300 dimensions across their business.
Example Data Columns:
- Timestamp (categorical)
- Store # (categorical)
- State (categorical)
- Region (categorical)
- Market (categorical)
- (293+ other dimensions or columns)
- Product Name (categorical)
- Total Sales (numerical)
Questions trying to solve for:
- What is the best model to identify anomalies in a dataset like this? Isolation Forests look promising.
- Should I build different models for different types of anomalies? (i.e. positive/negative slopes, new sales that escalate from 0 to something, spikes, etc.)
- If a model is built, what is the best way to not only identify an anomaly but identify the combinations/columns leading to the anomaly (i.e. root cause)? Brute force?
- If combinations are identified, what is the best way to ensure the combinations have enough data to be statistically relevant? Or what if I identified the anomaly in another combination already?
Given the high number of dimensions, I'm struggling to identify the best type of model to train for anomalies, how best to use the model to identify root causes of anomalies, and ensure the root causes are statistically relevant enough to review. Looking across 300 dimensions is not feasible to review manually and the data isn't always normal like products coming off a conveyor belt. I am trying to eliminate many of the dimensions but it seems difficult to narrow this down to 1 or 2 dimensions.
Appreciate any recommendations or suggestions on how best to approach the problem. Any relevant books is also recommended. Thanks!