0
$\begingroup$

My company processes Data (I am only an intern). We primarily use Hadoop. We're starting to deploy spark in production. Currently we have have two jobs, we will choose just one to begin with spark. The tasks are:

  1. The first job does analysis of a large quantity of text to look for ERROR messages (grep).
  2. The second job does machine learning & calculate models prediction on some data with an iterative way.

My question is: Which one of the two jobs will benefit from SPARK the most?

SPARK relies on memory so I think that it is more suited to machine learning. The quantity of DATA isn't that large compared to the logs JOB. But I'm not sure. Can someone here help me if I neglected some piece of information?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I think second job will benefit more from spark than the first one. The reason is machine learning and predictive models often run multiple iterations on data.

As you have mentioned, spark is able to keep data in memory between two iterations while Hadoop MapReduce has to write and read data to file system.

Here is a good comparison of the two frameworks :

https://www.edureka.co/blog/apache-spark-vs-hadoop-mapreduce

enter image description here

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you Sir. Just what I needed. So hadoop isn't suited for a lot of small iteration ? $\endgroup$
    – Melchia
    Commented Jan 7, 2018 at 14:12
  • 1
    $\begingroup$ Yes! Because at the end of every iteration, Hadoop MapReduce writes data on HDFS (or other file system) and then loads it again for next iteration. Spark is more recent than Hadoop MapReduce and it has been a small revolution with this improvement. Project Apache Mahout (Machine learning on Hadoop) has really suffered from it and in just few weeks, there were no more contributors to the project. Spark MLLib and then Spark ML became more famous solutions. $\endgroup$
    – Theudbald
    Commented Jan 7, 2018 at 14:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.