5
$\begingroup$

I just started learning about Hadoop. From what I understand its primary strength is its ability to distribute a task across many nodes. Is there any benefit to using Hadoop with only a single node, other then the potential of expanding to more nodes in the future? What if you have no plans of ever having more then one node? Or is there an expectation that if you are going to use Hadoop you are also going to be setting up a cluster of many nodes?

$\endgroup$
4
$\begingroup$

If you are testing or building something that will eventually be run on a distributed cluster, yes it make sense. If you're playing around to learn, yes. Otherwise no I don't see much value.

$\endgroup$
2
$\begingroup$

Hadoop, HDFS etc. come at a substantial cost.

Depending on your application and code quality, the break even is usually somewhere between 10 and 100 nodes: a beginners Hadoop job may easily run 100x slower than an average non-Hadoop job, so you need this many nodes (and data!) to make up for the overhead.

This gets much worse when the non-Java APIs are used. The native layer of Hadoop is Java, and running e.g. a Python or R map-reduce job will usually require multiple serializations and deserializations of data into text and back. For simple map jobs such as the famous word-count example, this overhead is more than the actual task.

As a rule of thumb: if your data fits into main memory, don't put it on hadoop but process it locally. It may be much more effective to use multithreading or GPUs then. On the other hand, if your data is several TB, and you only need a tiny bit of that data, reading from multiple harddisks in parallel does offer advantages. That is why a lot of the use cases follow the ETL pattern. Scan a large amount of raw data, extract the required parts, transform them into the desired format, and load them into a different system for further processing. After that, the data usually is small enough to be processed with "conventional" tools. It's disappointing, but PCs have just a lot of RAM these days... and CPU+GPU power is plenty, too.

Also, benchmark. You'll be surprised how expensive the network and architecture overheads are.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.