0
$\begingroup$

I am trying to implement Anomaly Detection over a multivariate dataset having nominal and numerical predictors.

Dataset has following pattern: Data pattern

If we consider the below sample records, category_id, currency, and product_id are nominal predictors, whereas price is a numerical variable.

My model is able to identify the anomaly in the price for '_id=4' because the price range for different products for the particular combination of category_id-currency-product_id is between 10-500EUR.

But it is not able to identify anomalies for product_id=1, product_id=2 or product_id=3 as they have unusual category_id or currency or product_id.

Types of Anomalies

Correlation matrix for dataset:

Correlation matrix for dataset

I am applying 'Multiple Correspondance Analysis(MCA)' over nominal predictors and reducing these 3 predictors to 2 predictors.

Algorithm: I have tried with OC-SVM, Isolation Forest, and Autoencoders, but none perform as expected.

Some of the resources that I have checked but to no help: Unsupervised Anomaly Detection with Mixed Numeric and Categorical Data

Anomaly Detection/Novelty detection

I am not able to understand where I am making the mistake: Data pre-processing OR Algorithm OR something else?

I would highly appreciate any help here.

You can review my code here.

Improve this question
$\endgroup$
2
  • $\begingroup$ How are you encoding nominal variables? $\endgroup$
    – bonfab
    Jun 28, 2020 at 13:25
  • $\begingroup$ @bonfab Initially I tried OneHotEncoding but that's not helpful in long run as number of classes will only increase.e.g.diff. currencies. Then I used Multiple Correspondence Analysis (MCA) over the 3 nominal attributes. $\endgroup$
    – Dhaval Simaria
    Jun 28, 2020 at 13:28

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.