2
$\begingroup$

I am working on a hobby project where I have a data related to financial trades(such as stock trades). Now I want to analyse this data and extract possible suspicious patterns and trades. Since I am new to Data Science so I would love to get few suggestions regarding what are some effective algorithms(machine learning) that can be relevant to such a scenario. One category that I think is using a possible outlier detection algorithm but I am unable to determine which one to use. I would also appreciate few pointers related to what my approach should be.

$\endgroup$
3
$\begingroup$

I suggest , go for Anomaly detection:

Anomaly Detection is done assuming our data has a probability distribution(gaussian). We can plot data to see if thats the case, if not we can make it gaussian using log transforms. Gaussian distribution specifies the regions and probabilities of our data lying in those regions.

For example : replace original feature x -> Log(x) or feature x -> (x)^4/3 etc..

Also regarding the threshold value which decides outliers you can play with it and see that with Higher threshold you will be rejecting more entries and this might be required where doctors are trying to isolate cancer patients amongst many normal ones without taking any risk/chances. Again outliers here doesn't mean cancer patient but definitely worth a medical test. And you can set it to lower value if you are getting too many normal data flagged as outliers.

We have skewed data sets since we have more examples of one kind than the other. For example when we get air craft engine data we might just have data for few bad ones and mostly for good ones.Use of cross validation data is suggested.F1-Score is a pretty good metric to evaluate the performance of the algorithm.

To get a proper hold on this topic , I also suggest go through anomaly detection course videos by Andrew NG in machine learning on Coursera. Free course and very nicely made.


The well known techniques for the same are: Density-based techniques (k-nearest neighbor),Support vector Machines,Cluster Analysis Anomaly detection. People also use Fuzzy logic based techniques. And for python I suggest use libraries like scikit-learn and TensorFlow , they have superb implementations of usable machine learning algorithms.Heres a video link from a googler,giving general overview for the same :https://www.youtube.com/channel/UCwUtgW0JCPFjLbKx4VVr3bw

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the inputs. Can you mention some famous anomaly detection algorithms (maybe from the point of view of Java or Python) that seems relevant? $\endgroup$ – user2916886 Aug 3 '16 at 5:15
  • $\begingroup$ I have edited my answer to reflect the same. $\endgroup$ – Xeqtr Aug 3 '16 at 5:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.