0
$\begingroup$

Right now

I've been doing a bit of research because I'm quite new to Big Data world. Among several other tools or frameworks, I've read about Apache Hadoop and Python for data analysis.

Specifically, I've read that:

Hadoop: allows you to perform any task you want, if it fits the Map-Reduce paradigm. It can add concurrence using a cluster, etc.

SciPy: python based ecosystem for (in fact, a lot of things)...

So my question is

If I've data collected from an environment, and want to correlate the data, calculate means, extract conclusions, etc...

Is SciPy as suitable as Hadoop (or more) to do so?

I'm basically trying to choose between them in order to build a little example, so I wonder which one fits better, or may I be misunderstanding the uses of each one.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

I think you're quite confused.

Hadoop is a collection of software that contains a a distributed file system called HDFS. Essentially HDFS is a way to store data cross a cluster. You can access file stores as you would in a local file store (with some modification) and modify things via Java API. Furthermore, ON TOP OF the file system there exist a MapReduce engine that allows for distributive workflow.

Python on the other hand is a generic programming language that can be made to do a myriad of task such as build a web applciation, to generating reports and even peforming analytics.

SciPy is a package that can be used in conjunction with Python (and often numpy) as a way to perform common scientific task.

Truthfully, they focus on different paradigms. If you have LARGE DATA (ie terabytes worth of it), it might be worth wild to setup a hadoop cluster (ie multiple servers and racks) and use Java MapReduce, Hive, Pig or Spark (of which there is a python version) to do analytics.

If your data is small or you only have one computer, then it probably makes sense to just use python instead of adding the overhead of setting up hadoop.

Edit: Made correction via comment.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Though this answer has been rated as the best one, it has to be said that Hadoop is not a distributed file system. Hadoop is a collection of software that performs MapReduce. The distributed file system of Hadoop is HDFS (Hadoop Distributed File System) $\endgroup$
    – ignatius
    May 10, 2018 at 6:39
  • $\begingroup$ Added correction $\endgroup$
    – Tophat
    May 14, 2018 at 12:58
1
$\begingroup$

If you have the data on your local machine, I guess, although quite large, you can perform the operations you want avoiding the Map-Reduce philosophy, thus go for python and scipy.

The first thing you should bare is wether you want to perform Map-Reduce. Map-Reduce is a way of manipulating very large amounts of data. Taking it very simply, Map-Reduce divides the data into smaller chunks and the analysis is done over each chunk in parallel (Mapping stage). When all the chunks have been analyzed, the partial results have to be collected and summarized (Reducing-stage). Typically, Map-reduce/Hadoop is performed in both distributed data and computing systems (Amazon S3, etc... One important task of the mapping stage is to orchestrate how the different chunks of data are shared over the nodes that conform the distributed system). Hadoop is a software framework written in Java for doing Map-Reduce.

If you decided your needs fit the Map-Reduce, go for Hadoop-Java. It is possible, though, to run Hadoop operations also in python.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.