Take the 2-minute tour ×
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It's 100% free, no registration required.

An aspiring data scientist here. I don't know anything about Hadoop, but as I have been reading about Data Science and Big Data, I see a lot of talk about Hadoop. Is it absolutely necessary to learn Hadoop to be a Data Scientist?

share|improve this question
1  
This class of questions is being discussed on meta. You can voice your opinion on this meta post. –  AsheeshR Jun 11 '14 at 2:19
    
It is not absolutely necessary. It is just one of the tools. What is necessary is an understanding of statistics and linear algebra. The choice of tool is secondary. –  Victor Jul 9 at 2:01
    
Look at this free e-book and it tries to answer your question.oreilly.com/data/free/files/analyzing-the-analyzers.pdf –  Espanta Jul 13 at 1:49
    
I have a similar question on IBM Watson Analytics,Google's Bigquery and other cloud based analytics are this technologies better then Hadoop and spark .....I am just starting to learn Hadoop and spark and do I really need to learn Hadoop and spark to do big data analytics –  DrPurshottam KH Jul 24 at 21:29

10 Answers 10

up vote 24 down vote accepted

Different people use different tools for different things. Terms like Data Science are generic for a reason. A data scientist could spend an entire career without having to learn a particular tool like hadoop. Hadoop is widely used, but it is not the only platform that is capable of managing and manipulating data, even large scale data.

I would say that a data scientist should be familiar with concepts like MapReduce, distributed systems, distributed file systems, and the like, but I wouldn't judge someone for not knowing about such things.

It's a big field. There is a sea of knowledge and most people are capable of learning and being an expert in a single drop. The key to being a scientist is having the desire to learn and the motivation to know that which you don't already know.

As an example: I could hand the right person a hundred structured CSV files containing information about classroom performance in one particular class over a decade. A data scientist would be able to spend a year gleaning insights from the data without ever needing to spread computation across multiple machines. You could apply machine learning algorithms, analyze it using visualizations, combine it with external data about the region, ethnic makeup, changes to environment over time, political information, weather patterns, etc. All of that would be "data science" in my opinion. It might take something like hadoop to test and apply anything you learned to data comprising an entire country of students rather than just a classroom, but that final step doesn't necessarily make someone a data scientist. And not taking that final step doesn't necessarily disqualify someone from being a data scientist.

share|improve this answer
1  
Thank you for the detailed answer. That is relaxing! –  Pensu Jun 10 '14 at 8:29

As a former Hadoop engineer, it is not needed but it helps. Hadoop is just one system - the most common system, based on Java, and a ecosystem of products, which apply a particular technique "Map/Reduce" to obtain results in a timely manner. Hadoop is not used at Google, though I assure you they use big data analytics. Google uses their own systems, developed in C++. In fact, Hadoop was created as a result of Google publishing their Map/Reduce and BigTable (HBase in Hadoop) white papers.

Data scientists will interface with hadoop engineers, though at smaller places you may be required to wear both hats. If you are strictly a data scientist, then whatever you use for your analytics, R, Excel, Tableau, etc, will operate only on a small subset, then will need to be converted to run against the full data set involving hadoop.

share|improve this answer

You have to first make it clear what do you mean by "learn Hadoop". If you mean using Hadoop, such as learning to program in MapReduce, then most probably it is a good idea. But fundamental knowledge (database, machine learning, statistics) may play a bigger role as time goes on.

share|improve this answer
    
Apparently most of the people are using Hadoop for analytics. What I am thinking is do I need something like that or knowledge about database, ML, statistics is enough? –  Pensu Jun 12 '14 at 18:26

Yes, you should learn a platform that is capable of dissecting your problem as a data parallel problem. Hadoop is one. For your simple needs (design patterns like counting, aggregation, filtering etc.) you need Hadoop and for more complex Machine Learning stuff like doing some Bayesian, SVM you need Mahout which in turn needs Hadoop (Now Apache Spark) to solve your problem using a data-parallel approach.

So Hadoop is a good platform to learn and really important for your batch processing needs. Not only Hadoop but you also need to know Spark (Mahout runs it's algorithms utilizing Spark) & Twitter Storm (for your real time analytics needs). This list will continue and evolve so if you are good with the building blocks (Distributed Computing, Data-Parallel Problems and so on) and know how one such platform (say Hadoop) operates you will fairly quickly be up to speed on others.

share|improve this answer

It strongly depends on the environment/company you are working with. In my eyes there is a "big data" hype at the moment and a lot of companies try to enter the field with hadoop based solutions - what makes hadoop also a buzzword but its not always the best solution.

In my mind, a good Data Scientist should be able to ask the right questions and keep on asking again until its clear whats really needed. Than a good DataScientist - of course - needs to know how to address the problem (or at least know someone who can). Otherwise your stakeholder could be frustrated :-)

So, i would say its not absolutely necessary to learn Hadoop.

share|improve this answer

You should learn Hadoop if you want to be work as data scientist, but maybe before starting with Hadoop you should read something about ETL or Big Data... this book could be a good starting point: http://www.amazon.com/Big-Data-Principles-practices-scalable/dp/1617290343

Hope it helps and good luck!

share|improve this answer

You can apply data science techniques to data on one machine so the answer to the question as the OP phrased it, is no.

share|improve this answer

It depends on your employer. Many stipulate that you know it, especially if the job involves "big data", while others will let you learn on the job or not care. Take a look at the job boards and see for yourself!

share|improve this answer

Data Science is a field demanding a variety of skills. Having knowledge of Hadoop is one of them. The main tasks of a Data Scientist include:

  1. Gathering data from different resources.
  2. Cleaning and pre-processing the data.
  3. Studying statistical properties of the data.
  4. Using Machine Learning techniques to do forecasting and derive insights from the data.
  5. Communicating the results to decision makers in an easy to understand way.

Out of the above points knowledge of Hadoop is useful for points 1,2 and 3, but you also need to have strong mathematical/statistical background and strong knowledge of Computational techniques to work in data science field. Also Hadoop is not the only framework that is being used in Data Science. Big Data ecosystem has a range of frameworks, each specific to a particular use case. This article gives introductory material regarding major Big Data frameworks that could be used in Data Science:

http://www.codophile.com/big-data-frameworks-every-programmer-should-know/

share|improve this answer

I do think Leaning Hadoop framework (hard way) is not a requirement of being a Data Scientist. General knowledge on all big data platforms is essential. I will suggest to know concept on it and only part need from Hadoop is the MapReduce http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html

A Data Scientist does not build cluster, administer ... is just make "magic" with data and does not care where is coming from. The term "Hadoop" has come to refer not just to the base modules above, but also to the "ecosystem", or collection of additional software packages that can be installed on top of or alongside Hadoop, such as Apache Pig, Apache Hive, Apache HBase, Apache Spark, and others.

Most important is the Programing language, math and statistics for working with data (you'll need to find a way to connect with data and move forward). I wish I had somebody to point me to the concept and do not spend weeks on learning framework and build from scratch nodes and clusters, because that part is Administrator role and not Data Engineer or Data Scientist. Also one thing: all are changing and evolving but math, programing, statistics are still the requirements.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.