Outlier detection by unsupervised algorithm: Fraud Detection

Question

I have set of 300,000 set of rows with credit card transactions and my job is to find outliers (suspicious transactions) in those dataset.

I have created around 5 features (All continuous data, with 1 column as transaction id)

I need to return list of all transaction id,which looks suspicious

What have I tried

I have tried using K means algorithm, but it does does not fit my laptop's memory (8 GB) and it crashes.

I wanted to try 1 class SVM, but I do not see any good tutorial to get me started. I tried scikitlearn official tutorial, but it already has outliers added to it and they are just plotting it.

http://scikit-learn.org/stable/auto_examples/svm/plot_oneclass.html#sphx-glr-auto-examples-svm-plot-oneclass-py

http://scikit-learn.org/stable/auto_examples/covariance/plot_outlier_detection.html#sphx-glr-auto-examples-covariance-plot-outlier-detection-py

How do I automatically detect the outliers and return those observations ?

Answer 1

First of all, start with a subset until you know what you are doing. There is no use in waiting for hours for a result that doesn't work, or to run out of memory, or to optimize, just to find out it does not work.

Secondly, makes sure your preprocessing is very very well done. Bad preprocessing will hurt your algorithms.

From my experience, one-class SVM does not work well. It assumes all your training data is 'normal' class (no outliers), and this a representative sample of all normal data. Secondly,.you need to tune kernel parameters, but how.can you do so without labeled anomalies?

Instead, I'd try knn outlier detection, LOF and LoOP.

But for these you need to make sure your distance is a very good measure of similarity. If you don't preprocess well, distance does not work, and then nearest-neighbor methods don't work either.

Answer 2

Ok, this is a bit late, but two points which will hopefully be of help for someone in the future.

1. The labels that you are looking for should be returned by clf.predict (see also in the tutorial that you posted)

2. As can be seen in this quesion Why won't my SVM learn a sequence of repeated elements your testset actually must contain outliers. This contradicts the statement in scikit-learn that they are doing novelty detection. To me, the given link proves that the method does not (not that this would be a problem, it is just important to know).