3
$\begingroup$

I have set of 300,000 set of rows with credit card transactions and my job is to find outliers (suspicious transactions) in those dataset.

I have created around 5 features (All continuous data, with 1 column as transaction id)

I need to return list of all transaction id,which looks suspicious

What have I tried

I have tried using K means algorithm, but it does does not fit my laptop's memory (8 GB) and it crashes.

I wanted to try 1 class SVM, but I do not see any good tutorial to get me started. I tried scikitlearn official tutorial, but it already has outliers added to it and they are just plotting it.

http://scikit-learn.org/stable/auto_examples/svm/plot_oneclass.html#sphx-glr-auto-examples-svm-plot-oneclass-py

http://scikit-learn.org/stable/auto_examples/covariance/plot_outlier_detection.html#sphx-glr-auto-examples-covariance-plot-outlier-detection-py

How do I automatically detect the outliers and return those observations ?

$\endgroup$
  • $\begingroup$ Have you looked at kernel density estimation? $\endgroup$ – D.W. Mar 6 '17 at 3:50
  • $\begingroup$ Boxplots might help, 3 STD rule also.. $\endgroup$ – Aditya Feb 28 '18 at 10:38
8
$\begingroup$

First of all, start with a subset until you know what you are doing. There is no use in waiting for hours for a result that doesn't work, or to run out of memory, or to optimize, just to find out it does not work.

Secondly, makes sure your preprocessing is very very well done. Bad preprocessing will hurt your algorithms.

From my experience, one-class SVM does not work well. It assumes all your training data is 'normal' class (no outliers), and this a representative sample of all normal data. Secondly,.you need to tune kernel parameters, but how.can you do so without labeled anomalies?

Instead, I'd try knn outlier detection, LOF and LoOP.

But for these you need to make sure your distance is a very good measure of similarity. If you don't preprocess well, distance does not work, and then nearest-neighbor methods don't work either.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I don't agree that training data needs be normal for one class SVM classifiers. SVM is a distance based classifier and the distribution of the dataset should not matter. $\endgroup$ – FrankZhu Sep 21 '18 at 14:07
  • $\begingroup$ Normal as in "not outlier", not as in "normal distributed". $\endgroup$ – Has QUIT--Anony-Mousse Sep 21 '18 at 19:28
  • $\begingroup$ oh, i see you point now. $\endgroup$ – FrankZhu Sep 22 '18 at 13:30
1
$\begingroup$

Ok, this is a bit late, but two points which will hopefully be of help for someone in the future.

  1. The labels that you are looking for should be returned by clf.predict (see also in the tutorial that you posted)

  2. As can be seen in this quesion Why won't my SVM learn a sequence of repeated elements your testset actually must contain outliers. This contradicts the statement in scikit-learn that they are doing novelty detection. To me, the given link proves that the method does not (not that this would be a problem, it is just important to know).

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.