anomaly detection - conceptual help on machine learning

Question

I am working on Anomaly detection model problem for a finance data set - set of card transactions. My team member suggested an idea that " First train the model with normal instances of data and to the trained model pass the anomalous data during testing to see whether the model will be able to detect the anomaly or not." Is it possible like this with any Machine Learning algorithms. If so, can anybody suggest what algorithms can be used for this idea?

An additional question: How about the situation when there is no properly labelled fraudulent data and we have just transactional information. Does semi-supervised learning works? I do not have experience in this type of ML approach. Can anybody suggest good resources for this and also suggest can Semi-supervised learning or hybrid supervised learning works for anomaly/ fraud detection?

Answer 1

There is two options:

You have a label

Then what you probabibly have is a binary classification problem, probably unbalanced. If this happens follow this https://towardsdatascience.com/practical-tips-for-class-imbalance-in-binary-classification-6ee29bcdb8a7

You have no label

Then you have an unsupervised learning problem, in finance Isolation Forest is really used. If this is the case have a look at this https://towardsdatascience.com/outlier-detection-with-isolation-forest-3d190448d45e

Answer 2

There are several choices:

• PDF estimation: the normal cases would be close to nodes and anomalies would be defined as outliers (you can use kernel methods but these do not fare well in high dimension) In this setting you estimate the normal case pdf and then use this pdf to predict if a new point is an anomaly: anomalies are “far away” from normal cases. How far is an hyperparameter that you need to optimise.

• use binary classification. There you have to use both cases to fit your model. Among popular models to do so: logistic regression, Support vector machines our neural networks.

Have a look at Andrew NG’s Coursera ML class he has a section on anomaly detection that is very well explained.

Answer 3

How about Multivariate Gaussian?

Multivariate Gaussian finds means of all features, as well as covariance matrix of all features (you can use the normal instances for that). It uses this information to identify "how normal" a data point is. If the score is below a given threshold, the data point is an anomaly. You can supply it with known normal data points and known anomalies to test how it works.

*Normal data points that you use to test should not be part of normal data data points that you used to find means and covariance matrix.