-1
$\begingroup$

I don't need different between Explain why Spark is considered “in-memory” compared to Hadoop.

$\endgroup$
1
  • 3
    $\begingroup$ can you provide the reference for this claim? To my knowledge both use storage other than memory $\endgroup$ – Nikos M. May 4 at 16:42
1
$\begingroup$

You probably mean "MapReduce" by "Hadoop". MapReduce is an older model where each operation took input from storage, computed, wrote results to storage. Of course, many real-world tasks involve lots of operations - read, filter, transform, aggregate, etc. Expressing those in MapReduce would be difficult, and what's more, every single piece of the operation would read and write data, which is slow.

Spark is a higher-level framework where the operations can be combined and optimized first before execution. So a bunch of simple operations on data could all happen at once to data, while it's 'in memory', without any need to persist intermediate results.

This is also said of Spark because it can easily persist data sets in memory if desired by caching them. This easy access to data in memory is also one reason Spark can be faster.

Of course, nobody really used MapReduce for basic SQL-like operations. They'd have used Hive or similar. Hive also had similar optimizations and was likewise, to some degree, 'in memory'. However Hive was SQL-only, whereas Spark can support distributed SQL operations, user code, etc. It had the flexibility of MapReduce with much less overhead for developers, but could also support high-throughput SQL like Hive.

$\endgroup$
2
  • $\begingroup$ Although I understand trying to guess what the OP means by his question and providing a good answer to that. I would rather wait for OP to clarify and provide the reference for the claim he makes $\endgroup$ – Nikos M. May 5 at 18:18
  • $\begingroup$ Spark most certainly uses storage too, so I think that can't be it. I've heard this repeated enough that I can guess what the Q is. $\endgroup$ – Sean Owen May 5 at 19:40

Not the answer you're looking for? Browse other questions tagged or ask your own question.