4
$\begingroup$

I'm a data analyst/scientist working mostly with the Python open source stack i.e. Pandas, scikit-learn, matplotlib, seaborn etc. I want to expand my toolbox and learn a distributed computing framework.

Hadoop had created a lot of fuss in around 2013-2014. As per my limited knowledge on the subject, Apache Spark has improved upon Hadoop multi-fold in all aspects.

So, apart from the obvious case when you need to maintain legacy Hadoop applications, is there any reason as of today to prefer Hadoop over Apache Spark?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

At this point of time, if I had to start a project from scratch and I had to choose between Hadoop and Spark, I would certainly choose Spark over Hadoop. There are several reasons for this:

  • Spark is more efficient than Hadoop given the fact that processing in Spark is in-memory whereas Hadoop requires to store intermediate results in disk.
  • Spark provides dozens of different operations and it is not constrained to just Hadoop's map-reduce.|

However, I still think that there is value in learning Hadoop before trying to learn Spark, even if it is at a high level. These are some reasons:

  • You may came across some legacy applications/systems based on Hadoop technologies
  • It provides a gentle introduction to some of the concepts used in Spark
  • Spark is sometimes used in combination with some technologies in the Hadoop ecosystem, like Hive and HDFS (Spark does not incorporate a way of storing data, but it can fetch data from multiple sources, including HDFS, and a Spark cluster with HDFS based data storage is a usual combination).
Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks. (1) "some legacy applications/systems based on Hadoop technologies", do you mean some Hadoop technologies are outdated? Which are those Hadoop technologies? Which Hadoop technologies are still very relevant? I am starting to self learn Hadoop and Spark, so pardon me for my naive questions. (2) For learning Hadoop ecosystem and Spark, is it better to use a Hadoop distribution, or install Hadoop and other components directly, or something else? If using a Hadoop distribution, which one would you recommend Cloudera, Hordonworks, MapR (the first two owning companies have just merged)? $\endgroup$
    – Tim
    Commented Apr 1, 2019 at 14:36
  • $\begingroup$ Hadoop is in clear decline and being replaced by Spark and other technologies. However, some companies have services running with Hadoop that were built some time ago, when Hadoop was still very popular, and you may need to be able to maintain these services as part of your job role. Different components of Hadoop may be relevant for different people, but I would say that HDFS and MapReduce should suffice. From there you should be able to move on to Spark. $\endgroup$
    – Pablo Suau
    Commented Apr 4, 2019 at 14:27
  • $\begingroup$ In terms of suggested environments and distributions, I'd say that it depends on your resources and needs. I don't feel like the most qualified person to answer that question. Maybe you can create a new question at datascience.stackexchange.com? $\endgroup$
    – Pablo Suau
    Commented Apr 4, 2019 at 14:29
  • $\begingroup$ Thanks. So I will learn HDFS, MapRduce and then Spark. Yes, I don't want to study outdated/replaced technologies. Do you recommend to learn Yarn, Pig, Hive, HBase, and Storm, or do they have replacements? $\endgroup$
    – Tim
    Commented Apr 4, 2019 at 14:37
  • $\begingroup$ From all of those, I'd only recommend Hive. In fact, I'd take a look at it before starting with Spark. $\endgroup$
    – Pablo Suau
    Commented Apr 4, 2019 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.