When dealing with anomaly detection in a dataset with a large number of categories, it's important to consider the nature of your data and choose an appropriate algorithm. Isolation Forest is a popular choice for anomaly detection, but its performance can vary depending on the characteristics of your dataset, also traditional algorithms may not perform well due to the ordinal encoding issue you mentioned. In such cases, you can consider using more advanced techniques that can handle categorical variables effectively.
In your case, where you have categorical features such as "Parcel," "From," and "To," encoding them using ordinal relationships may not capture the true nature of the data. Instead, you should consider using one-hot encoding, which creates binary features for each category. This approach allows the algorithm to consider the categorical features individually rather than assuming a specific ordering.
Apart from Isolation Forest & OHE, you might also consider other algorithms that are suitable for anomaly detection in high-dimensional data, such as:
Entity Embeddings: Instead of one-hot encoding, you can use entity embeddings to represent your categorical variables. Entity embeddings are low-dimensional vector representations that capture the semantic relationships between categories. By training a neural network to learn these embeddings, you can create meaningful representations of your categorical features. You can then feed these embeddings into an anomaly detection algorithm such as an autoencoder or an outlier detection model like the Local Outlier Factor (LOF).
Supervised Anomaly Detection: If you have labeled anomalies in your dataset, you can use supervised anomaly detection techniques. In this approach, you train a model to classify parcels as normal or anomalous based on the given routes. Techniques like Support Vector Machines (SVMs), Random Forests, or Gradient Boosting models can be trained on labeled data to identify anomalies based on the patterns observed in the features.
Deep Autoencoders: Deep autoencoders are neural networks that are trained to reconstruct their input. By encoding the input into a lower-dimensional representation and then decoding it back to the original space, the autoencoder learns the underlying patterns and structures in the data. Anomalies can be identified based on the reconstruction error, where higher errors indicate samples that deviate significantly from the learned patterns.
Gaussian Mixture Models (GMM): GMM assumes that the data is generated from a mixture of Gaussian distributions. Anomalies can be identified based on low likelihood values.
