0
$\begingroup$

I know that Spark enhances performance relative to mapreduce by doing in-memory computations. But, since the caching is explicitly decided by the programmer, one can also proceed without doing that. I believe even in such cases, Spark is 10x faster than map reduce. What features in the framework make this possible?

$\endgroup$
4
$\begingroup$

No, this is not in general true. For a map-only job, or map and reduce, MapReduce is a bit faster. It's more optimized for this pattern and a lot easier to scale / tune. However, the problem is that few problems are just one or two operations. Once you have a chain of them to execute, considering the entire DAG and execution plan is a win even if memory is not used for persistence.

Developer productivity also related to efficiency. If I don't have to spend my time writing lower-level code then I can spend more time designing a more sophisticated distributed application that could be faster.

You can see this in Crunch, for example, which gives you a high-level language similar to Spark Java on MapReduce, and I'd say you can write more efficient Crunch jobs in a fixed time than M/R.

$\endgroup$
1
  • $\begingroup$ So the DAG is the contributing factor then? Just like the way Tez speeds up Hive $\endgroup$
    – Vinay
    Nov 28 '15 at 16:02
1
$\begingroup$

When you script out steps for spark on an RDD, it does not begin executing the operations until the data needs to be accessed. Instead of actually manipulating the data, spark builds a graph of how to go from the data, to the desired result set, and will not store intermediate data. For example, if you enter the following into a shell

my_rdd = sc.parallelize(some_data)
mapped_data = my_rdd.map(...)
filtered_data = mapped_data.filter(...)
filtered_data.take(3)

you'll notice that when you enter lines 2 and 3, no execution occurs. This is because spark isn't creating mapped_data or filtered_data, but rather, is building a graph of how to get from my_rdd to filtered data. The only reason that you would really write it in this way, is if you need to access mapped_data at a later state, in which case you would want to use a cache.

$\endgroup$
4
  • $\begingroup$ Yes it uses lineage for fault tolerance and lazily evaluates. So this saving causes faster execution? $\endgroup$
    – Vinay
    Nov 27 '15 at 21:51
  • $\begingroup$ It can; learning the right places to cache results is a bit of an art when it comes to spark programing. I've been able to speed up programs by a good amount of time by caching in strategic points in time. $\endgroup$ Nov 27 '15 at 21:56
  • $\begingroup$ So just to sum up, the 10x speed that Spark achieves on disk compared to Map reduce is due to lazy evaluation? $\endgroup$
    – Vinay
    Nov 27 '15 at 22:07
  • 1
    $\begingroup$ No, it's just a part of it. The majority of the speed up is the in memory performance aspect you noted earlier. Caching is stored in memory as well, as opposed to map-reduce which actually writes to disk between stages (and is the biggest bottleneck). $\endgroup$ Nov 27 '15 at 23:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.