I have read a lot about Apache Spark. About RDD, DataFrames etc etc. But I haven't come across a simple explanation as to why and where should we be using Apache Spark? If my primary goal is machine/deep learning when should be the case that it be must to use Apache Spark?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
If you are just starting out, it is pretty unlikely that you come across such big datasets that require massive parallelization either in computing power or data storage means.
When you start pushing the limits of what a single machine can do, that's when you deploy a big data solution like Spark and Apache HDFS storage and start writing parallelizable models that scale well with bigger data.
In a small scale it might be actually way slower to run some jobs at Spark, because it adds some overhead to the calculations. Spark is best used for large enough, well parallelizable and iterative problems.
It is also worth mentioning that it requires some patience to configure the whole cluster to work well.
$\begingroup$
$\endgroup$
It depends. If your data is small and can be computed via online methods or within memory of a decent server, you'll probably be ok just using normal machine learning libraries like sk-learn or shogun.
However, in the real world, companies collect millions of data points with thousands of features. Holding this in memory becomes impossible or computationally impossible to run on a single machine.
Prior to Spark, and after mid 2000's, we handled this issue by writing MapReduce jobs and implementing distributive algorithms ie methods that could be ran across hundreds of machines and then combined again to get the same result. The problem was that MapReduce had a big issue with iterative algorithms, streaming data or being interactive.
Spark was introduced to solve this problem by the use of RDD and lineage. So essentially, you should use spark if you have a lot of data and you want to apply some iterative algorithm, stream data or have an interactive feel with the application.
Otherwise, you'll probably be ok with traditional MR or if your data is small using python and sklearn. If you're new to ML, I would recommend sticking with python and sklearn for the time being until you become comfortable with the algorithms there and framework.
