0
$\begingroup$

Looking for some advice. I am working on an Anomaly detection problem, I am looking at parcels being transported from A-B and want to identify which parcels are considered anomalies for given routes. My dataset contains millions of records something like the following

Parcel From To
TOYS US Spain
TOYS US Spain
TOYS US Spain
CARS US Spain
CARS US Spain
CARS US Spain
TOYS US JAPAN

After some googling, I have tried to use Isolation Forest but I seem to be getting random results.

I suspect that this is due to the encoding of my categories as ordinal relationships are being created between the encoded values. Is there a better algo that I should be using or any pointers that you can give?

$\endgroup$
1
  • $\begingroup$ Is there other data is associated with the records than Parcel,From,To ? $\endgroup$
    – Jon Nordby
    May 28, 2023 at 12:38

1 Answer 1

1
$\begingroup$

When dealing with anomaly detection in a dataset with a large number of categories, it's important to consider the nature of your data and choose an appropriate algorithm. Isolation Forest is a popular choice for anomaly detection, but its performance can vary depending on the characteristics of your dataset, also traditional algorithms may not perform well due to the ordinal encoding issue you mentioned. In such cases, you can consider using more advanced techniques that can handle categorical variables effectively.

In your case, where you have categorical features such as "Parcel," "From," and "To," encoding them using ordinal relationships may not capture the true nature of the data. Instead, you should consider using one-hot encoding, which creates binary features for each category. This approach allows the algorithm to consider the categorical features individually rather than assuming a specific ordering.

Apart from Isolation Forest & OHE, you might also consider other algorithms that are suitable for anomaly detection in high-dimensional data, such as:

  1. Entity Embeddings: Instead of one-hot encoding, you can use entity embeddings to represent your categorical variables. Entity embeddings are low-dimensional vector representations that capture the semantic relationships between categories. By training a neural network to learn these embeddings, you can create meaningful representations of your categorical features. You can then feed these embeddings into an anomaly detection algorithm such as an autoencoder or an outlier detection model like the Local Outlier Factor (LOF).

  2. Supervised Anomaly Detection: If you have labeled anomalies in your dataset, you can use supervised anomaly detection techniques. In this approach, you train a model to classify parcels as normal or anomalous based on the given routes. Techniques like Support Vector Machines (SVMs), Random Forests, or Gradient Boosting models can be trained on labeled data to identify anomalies based on the patterns observed in the features.

  3. Deep Autoencoders: Deep autoencoders are neural networks that are trained to reconstruct their input. By encoding the input into a lower-dimensional representation and then decoding it back to the original space, the autoencoder learns the underlying patterns and structures in the data. Anomalies can be identified based on the reconstruction error, where higher errors indicate samples that deviate significantly from the learned patterns.

  4. Gaussian Mixture Models (GMM): GMM assumes that the data is generated from a mixture of Gaussian distributions. Anomalies can be identified based on low likelihood values.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.