2
$\begingroup$

Apache Hadoop was once considered one of the tools you should have as a Data Scientist. Around 2012 to 2014, it gained the popularity steeply among Data Scientists, and seemed to be considered one of the toolsets on the same boat as Python, R, and SQL.

But these days I have not heard its story. In fact, the Google Trend also agrees with me, that the popularity is peak at May, 2015.

Why has Hadoop failed to become one of the necessary tools as a Data Scientist, like Python or R?

$\endgroup$
  • 2
    $\begingroup$ Because better options (with Spark at the forefront) sprang up to meet use cases (chiefly machine learning) not covered by the map-reduce paradigm. Hadoop is for dumb but big ETL tasks now. $\endgroup$ – Emre May 30 '17 at 6:28
3
$\begingroup$

I wouldn't say Hadoop failed to become popular rather I would say, it is still the base of any production big data system.

Python or R are handy at the beginning when you just need to try out things but when it comes to putting things in production, Hadoop is the way to go. It does not directly provide any data scientist tools what it does provide is the base where data would be stored, processed and can be used to apply machine learning algorithms using Spark.

So to summarize, I see Hadoop being used in the following contexts

  1. For Data Storage in Production Systems
  2. Hadoop YARN as a cluster manager for tools like Spark, Flink
  3. If someone still wants to code in R/Python/SAS - Hadoop being used as backend.

Hope this helps.

$\endgroup$
3
$\begingroup$

You are comparing apples with oranges. Hadoop is one of the backend of big data platform and Python/R are programming languages which are used built prediction models and data pipeline. Hadoop still can be used as data storage however more robust and faster distributed data storage frameworks are gaining popularity such as Apache Spark hence Hadoop lost its charm.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.