I am a newbie to Machine Learning and I have to do feature extraction for a banking application to detect / predict fraudulent transactions.

I have come across few articles on Feature Extraction using Scipy / Sci-kit learn in Python but I want to know is there any way to do the same in Apache Spark using some packages or modules within Spark

Any ideas that enable me to use Pyspark is also welcomed here

Appreciating your thoughts on this. Thanks in advance

  • $\begingroup$ Welcome to DataScience.SE! Learn to do it with pandas first; it uses the same paradigm. $\endgroup$
    – Emre
    Aug 18 '17 at 6:56
  • $\begingroup$ Thanks @Emre but it would be good if you please elaborate your thoughts in detail on how to do Feature Extraction / Selection with Pandas $\endgroup$
    – JKC
    Aug 18 '17 at 8:59
  • $\begingroup$ Do you know what feature extraction is? $\endgroup$
    – CalZ
    Aug 18 '17 at 12:57
  • $\begingroup$ Yes @CalZ theoretically $\endgroup$
    – JKC
    Aug 18 '17 at 14:21
  • $\begingroup$ Say you have a data set with a date (MM/DD/YYYY) in it. Feature engineering would be adding a column to that for day of the week (Sunday to Saturday) so that you can detect activity on a weekend vs weekday. Google how to "add a column in pandas data frame" and then how to do it with a Spark data frame. $\endgroup$
    – CalZ
    Aug 18 '17 at 14:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.