46

Different people use different tools for different things. Terms like Data Science are generic for a reason. A data scientist could spend an entire career without having to learn a particular tool like hadoop. Hadoop is widely used, but it is not the only platform that is capable of managing and manipulating data, even large scale data. I would say that ...


43

Hadoop means HDFS, YARN, MapReduce, and a lot of other things. Do you mean Spark vs MapReduce? Because Spark runs on/with Hadoop, which is rather the point. The primary reason to use Spark is for speed, and this comes from the fact that its execution can keep data in memory between stages rather than always persist back to HDFS after a Map or Reduce. This ...


16

Hadoop is not a database, hadoop is an entire ecosystem. Most people will refer to mapreduce jobs while talking about hadoop. A mapreduce job splits big datasets in some little chunks of data and spread them over a cluster of nodes to get proceed. In the end the result from each node will be put together again as one dataset. Let's assume you load into ...


13

MapReduce: A fault tolerant distributed computational framework. MapReduce allows you to operate over huge amounts of data- with a lot of work put in to prevent failure due to hardware. MapReduce is a poor choice for computing results on the fly because it is slow. (A typical MapReduce job takes on the order of minutes or hours, not microseconds) A ...


13

Let's first split it into parts. Data Science is about making knowledge from raw data. It uses machine learning, statistics and other fields to simplify (or even automate) decision making. Data science techniques may work with any data size, but more data means better predictions and thus more precise decisions. Hadoop is a common name for a set of tools ...


12

tl;dr: They markedly differ in many aspects and I can't think Redshift will replace Hadoop. -Function You can't run anything other than SQL on Redshift. Perhaps most importantly, you can't run any type of custom functions on Redshift. In Hadoop you can, using many languages (Java, Python, Ruby.. you name it). For example, NLP in Hadoop is easy, while it's ...


10

As Konstantin has pointed, R performs all its computation in the system's memory i.e. RAM. Hence, RAM capacity is a very important constraint for computation intensive operations in R. Overcoming this constraint, data is being stored these days in HDFS systems, where data isn't loaded onto memory and program is run instead, program goes to the data and ...


9

As a former Hadoop engineer, it is not needed but it helps. Hadoop is just one system - the most common system, based on Java, and a ecosystem of products, which apply a particular technique "Map/Reduce" to obtain results in a timely manner. Hadoop is not used at Google, though I assure you they use big data analytics. Google uses their own systems, ...


9

This is kind of like asking about the tradeoffs between frying pan and your drawer of silverware. They are not two things you compare, really. You might use them together as part of a larger project. Hadoop itself is not one thing, but a name for a federation of services, like HDFS, Hive, HBase, MapReduce, etc. Storm is something you use with some of these ...


8

R performs all computation in-memory so you can't perform operation on a dataset that is larger than available RAM amount. However there are some libraries that allow bigdata processing using R and one of popular libraries for bigdata processing like Hadoop.


7

You have to first make it clear what do you mean by "learn Hadoop". If you mean using Hadoop, such as learning to program in MapReduce, then most probably it is a good idea. But fundamental knowledge (database, machine learning, statistics) may play a bigger role as time goes on.


7

For your recommendation engine, if you've chosen to go by item similarity approach, then you can use Spark's RowMatrix datatype to achieve this task. Item similarity approach is just about creating a square matrix of items in your catalog (i.e. itemID X itemID), where each element of the matrix is the magnitude of similarity between and . This ...


6

HDFS Spark was built as an alternative to MapReduce and thus supports most of its functionality. In particular, it means that "Spark can create distributed datasets from any storage source supported by Hadoop, including your local file system, HDFS, Cassandra, HBase, Amazon S3, etc."1. For most common data sources (like HDFS or S3) Spark automatically ...


5

Current size limit for Amazon Redshift is 128 nodes or 2 PBs of compressed data. Might be circa 6PB uncompressed though mileage varies for compression. You can always let us know if you need more. anurag@aws (I run Amazon Redshift and Amazon EMR)


5

You can use map reduce algorithms in Hadoop without programming them in Java. It is called streaming and works like Linux piping. If you believe that you can port your functions to read and write to terminal, it should work nicely. Here is example blog post which shows how to use map reduce functions written in Python in Hadoop.


5

NoSQL is a way to store data that does not require there to be some sort of relation. The simplicity of its design and horizontal scale-ability, one way they store data is the key : value pair design. This lends itself to processing that is similar to Hadoop. The use of a NoSQL db really depends on the type of problem that one is after. Here is a good ...


5

Yes, this is a how-long-is-a-piece-of-string question. I think it's good to beware of over-engineering, while also making sure you engineer for where you think you'll be in a year. First I'd suggest you distinguish between processing and storage. Storm is a (stream) processing framework; NoSQL databases are a storage paradigm. These are not alternatives. ...


5

Indeed there are: Gradient Boosting is by construction sequential, so parallelization is not really possible Generalized Linear Models need all data at the same time, although technically you can parallelize some of the inner linear algebra nuts and bolts Support Vector Machines


4

There will definitely be a translation task at the end if you prototype using just mongo. When you run a MapReduce task on mongodb, it has the data source and structure built in. When you eventually convert to hadoop, your data structures might not look the same. You could leverage the mongodb-hadoop connector to access mongo data directly from within ...


4

You also can create a MongoDB-Hadoop connection.


4

Yes, you should learn a platform that is capable of dissecting your problem as a data parallel problem. Hadoop is one. For your simple needs (design patterns like counting, aggregation, filtering etc.) you need Hadoop and for more complex Machine Learning stuff like doing some Bayesian, SVM you need Mahout which in turn needs Hadoop (Now Apache Spark) to ...


4

Twitter uses Storm for real-time processing of data. Problems can happen with real-time data. Systems might go down. Data might be inadvertently processed twice. Network connections can be lost. A lot can happen in a real-time system. They use hadoop to reliably process historical data. I don't know specifics, but for instance, getting solid ...


4

There is a paper you should look into: MapReduce: Distributed Computing for Machine Learning They distinguish 3 classes of machine-learning problems that are reasonable to address with MapReduce: Single pass algorithms Iterative algorithms Query based algorithms They also give examples for each class.


4

Your job seems like a map-reduce job and hence might be good for Hadoop. Hadoop has a zoo of an ecosystem though. Hadoop is a distributed file system. It distributes data on a cluster and because this data is split up it can be analysed in parallel. Out of the box, Hadoop allows you to write map reduce jobs on the platform and this is why it might help ...


4

There's such an overwhelming amount of literature that with programming, databases, and Big Data I like to stick to the O'reilly series as my go-to source. O'reilly books are extremely popular in the industry and I've been very satisfied. A current version of Hadoop: The Definitive Guide, MapReduce Design Patterns, and Learning Spark might suit your ...


4

So if 4GB of RAM isn't sufficient, 1GB isn't going to be. That is really too little to run an HDFS namenode, a datanode, YARN, Spark driver alone, let alone leaving room for your workers. Much more reasonable is to simply run Spark locally on that instance without Hadoop at all. But I would question whether Spark is the right choice if you are definitely ...


4

Yes. There are examples on spark official document: https://spark.apache.org/examples.html Just put your HDFS file uri in your input file path as below (scala syntax). val file = spark.textFile("hdfs://train_data")


4

I would suggest that you can implement pretty much any kind of data processing you want in Map Reduce given enough time to code it, but the degree of parallelisation you will get will vary depending on what your data is and what you're doing to it. I can imagine a simple scenario where parallelisation would be reduced dramatically. For example, a simple ...


4

'SQL' on Hadoop is very much a thing, though I use quotes since it's probably more accurate to say it's SQL-like. Some options for bringing SQL-like capabilities to Hadoop include Hue, Hive/bee (Heading towards Stinger? So punny Apache), Impala, SparkSQL (probably not a great solution for a bank given the possibility of concurrency issues), among others (...


Only top voted, non community-wiki answers of a minimum length are eligible