Questions tagged [map-reduce]

MapReduce is a programming model for processing large data sets with a parallel, distributed algorithm on a cluster.

Filter by
Sorted by
Tagged with
0
votes
1answer
26 views

What is the MapReduce application master?

From Hadoop The Definitive Guide The whole process is illustrated in Figure 7-1. At the highest level, there are five independent entities: • The client, which submits the MapReduce ...
1
vote
0answers
23 views

How to read in all text files from UNIX bash directory in Cloudera's Python API

I'm still pretty new to Cloudera and using the UNIX environment. I have written a mapper that reads in .txt files from a directory in my Windows system, which works just fine. I read files in like ...
0
votes
1answer
36 views

How to generate list with out for loop in python [closed]

data frame S/N Type Number Capacity 1 Bike 2 5 2 Tempo 1 30 3 Truck-1 1 60 4 Truck-2 1 90 I would like to generate capacitylist = [5,5,30,60,90] Is it ...
0
votes
1answer
51 views

Which one of these tasks will benefit the most from SPARK?

My company processes Data (I am only an intern). We primarily use Hadoop. We're starting to deploy spark in production. Currently we have have two jobs, we will choose just one to begin with spark. ...
1
vote
0answers
107 views

How to multiply a “fat and short” matrix with a “tall and thin” matrix using MapReduce?

Assume that $A_{m \times n}$ and $B_{n \times k}$ are to be multiplied to get $C_{m \times k}$. Now if $n$ is too large for a single row $A_{j}$ of $A$ to fit in RAM (and similarly for columns of B) ...
1
vote
1answer
101 views

Suggestions on what patterns/analysis to derive from Airlines Big Data [closed]

I recently started learning Hadoop, I found this data set http://stat-computing.org/dataexpo/2009/the-data.html - (2009 data), I want some suggestions as what type of patterns or analysis can I do in ...
-1
votes
1answer
51 views

Is there a way of using multiple masters in a map reduce framework

If I have a single master, then if it fails, there is no one to tell the worker nodes what to do. Is there a way to repeat the Masters and making the workers require at least two masters saying the ...
2
votes
3answers
2k views

Why does map reduce have a shuffle step?

I'm looking at a diagram of map reduce where there is a map step, a shuffle step and then the reduce step. Why shuffle?
1
vote
1answer
368 views

Do something with the output of reducers

I am executing the job as: ...
5
votes
1answer
1k views

How to make k-means distributed?

After setting up a 2-noded Hadoop cluster, understanding Hadoop and Python and based on this naive implementation, I ended up with this code: ...
1
vote
0answers
165 views

Pig is not able to read the complete data

I am trying to load a huge dataset of around 3.4 TB with approximately 1.4 million files in Pig on Amazon EMR. The operations on the data are simple (JOIN and STORE), but the data is not getting ...
0
votes
2answers
299 views

Spark vs Map reduce

I know that Spark enhances performance relative to mapreduce by doing in-memory computations. But, since the caching is explicitly decided by the programmer, one can also proceed without doing that. I ...
2
votes
3answers
151 views

Can Hadoop be beneficial when data is in database tables and not in a file system

I work for a bank. Most of our data is in the form of database tables. Would we benefit by implementing Hadoop? I am of the impression that Hadoop is more for a Distributed File System (unstructured ...
4
votes
4answers
994 views

Can all statistical algorithms be parallelized using a Map Reduce framework

Is it correct to say that any statistical learning algorithm (linear/logistic regression, SVM, neural network, random forest) can be implemented inside a Map Reduce framework? Or are there ...
3
votes
1answer
65 views

Data produced as an output to Dumbo API of Python not getting distributed to all the nodes of cluster

On the node from which I run Dumbo commands, all the files produced as output are produced on the same node. For example, suppose there is a node having name hvs on which I ran the script: dumbo ...
2
votes
0answers
193 views

Time Complexity notation in Big Data platforms

I am redesigning some of the classical algorithms for Hadoop/MapReduce framework. I was wondering if there any established approach for denoting Big(O) kind of expressions to measure time complexity? ...
0
votes
1answer
188 views

MPI, MapReduce, or Spark for complex datasets and processing

I have 2 data files: the first one is a database, potentially very large; the second one contains queries I want to answer. My program pipeline is processing the database to get some information first,...
3
votes
0answers
33 views

Timing sequence in MapReduce

I'm running a test on MapReduce algorithm in different environments, like Hadoop and MongoDB, and using different types of data. What are the different methods or techniques to find out the execution ...
4
votes
3answers
294 views

What technologies are fastest at performing joins on large datasets?

By "large", I mean in the range of 100m to 10b rows. I'm currently using both Hadoop MapReduce and Amazon RedShift. MapReduce has been a little disappointing here. Redshift works very well if the ...
17
votes
3answers
5k views

Nearest neighbors search for very high dimensional data

I have a big sparse matrix of users and items they like (in the order of 1M users and 100K items, with a very low level of sparsity). I'm exploring ways in which I could perform kNN search on it. ...
8
votes
4answers
989 views

Data science and MapReduce programming model of Hadoop

What are the different classes of data science problems that can be solved using mapreduce programming model?
7
votes
1answer
1k views

Linear Regression in R Mapreduce(RHadoop)

I m new to RHadoop and also to RMR... I had an requirement to write a Mapreduce Job in R Mapreduce. I have Tried writing but While executing this it gives an Error. Tring to read the file from hdfs ...
3
votes
1answer
842 views

Difference Between Hadoop Mapreduce(Java) and RHadoop mapreduce

I understand Hadoop MapReduce and its features but I am confused about R MapReduce. One difference I have read is that R utilizes maximum RAM. So do perform parallel processing integrated R with ...
12
votes
3answers
3k views

Does Amazon RedShift replace Hadoop for ~1XTB data?

There is plenty of hype surrounding Hadoop and its eco-system. However, in practice, where many data sets are in the terabyte range, is it not more reasonable to use Amazon RedShift for querying ...
11
votes
3answers
653 views

Can map-reduce algorithms written for MongoDB be ported to Hadoop later?

In our company, we have a MongoDB database containing a lot of unstructured data, on which we need to run map-reduce algorithms to generate reports and other analyses. We have two approaches to select ...