All Questions
Tagged with map-reduce apache-hadoop
16 questions
1
vote
0
answers
120
views
Word count with map reduce
Suppose we use an input file that contains the following lyrics from a famous song:
We’re up all night till the sun
We’re up all night to get some
The input pairs for the Map phase will be the ...
- 151
1
vote
0
answers
42
views
Loading file into and out of HDFS via system call/cmd line vs using libhdfs
I am trying to implement a simple C/C++ program for the HDFS file system like word count, it takes a file from the input path puts it into HDFS (where it gets split), processed my map-reduce function ...
- 11
0
votes
2
answers
2k
views
What is the MapReduce application master?
From Hadoop The Definitive Guide
The whole process is illustrated in Figure 7-1. At the highest level,
there are five independent entities:
• The client, which submits the MapReduce ...
- 151
0
votes
1
answer
75
views
Which one of these tasks will benefit the most from SPARK?
My company processes Data (I am only an intern). We primarily use Hadoop. We're starting to deploy spark in production. Currently we have have two jobs, we will choose just one to begin with spark. ...
- 111
1
vote
0
answers
179
views
How to multiply a "fat and short" matrix with a "tall and thin" matrix using MapReduce?
Assume that $A_{m \times n}$ and $B_{n \times k}$ are to be multiplied to get $C_{m \times k}$. Now if $n$ is too large for a single row $A_{j}$ of $A$ to fit in RAM (and similarly for columns of B) ...
- 1,297
1
vote
1
answer
116
views
Suggestions on what patterns/analysis to derive from Airlines Big Data [closed]
I recently started learning Hadoop,
I found this data set http://stat-computing.org/dataexpo/2009/the-data.html - (2009 data),
I want some suggestions as what type of patterns or analysis can I do in ...
- 191
6
votes
1
answer
3k
views
How to make k-means distributed?
After setting up a 2-noded Hadoop cluster, understanding Hadoop and Python and based on this naive implementation, I ended up with this code:
...
- 291
1
vote
0
answers
265
views
Pig is not able to read the complete data
I am trying to load a huge dataset of around 3.4 TB with approximately 1.4 million files in Pig on Amazon EMR.
The operations on the data are simple (JOIN and STORE), but the data is not getting ...
- 373
2
votes
3
answers
273
views
Can Hadoop be beneficial when data is in database tables and not in a file system
I work for a bank. Most of our data is in the form of database tables. Would we benefit by implementing Hadoop? I am of the impression that Hadoop is more for a Distributed File System (unstructured ...
- 611
5
votes
4
answers
2k
views
Can all statistical algorithms be parallelized using a Map Reduce framework
Is it correct to say that any statistical learning algorithm (linear/logistic regression, SVM, neural network, random forest) can be implemented inside a Map Reduce framework? Or are there ...
- 611
4
votes
1
answer
86
views
Data produced as an output to Dumbo API of Python not getting distributed to all the nodes of cluster
On the node from which I run Dumbo commands, all the files produced as output are produced on the same node. For example, suppose there is a node having name hvs on which I ran the script:
dumbo ...
8
votes
4
answers
1k
views
Data science and MapReduce programming model of Hadoop
What are the different classes of data science problems that can be solved using mapreduce programming model?
- 369
7
votes
1
answer
1k
views
Linear Regression in R Mapreduce(RHadoop)
I m new to RHadoop and also to RMR...
I had an requirement to write a Mapreduce Job in R Mapreduce. I have Tried writing but While executing this it gives an Error.
Tring to read the file from hdfs
...
- 101
2
votes
1
answer
865
views
Difference Between Hadoop Mapreduce(Java) and RHadoop mapreduce
I understand Hadoop MapReduce and its features but I am confused about R MapReduce.
One difference I have read is that R utilizes maximum RAM. So do perform parallel processing integrated R with ...
- 101
12
votes
3
answers
3k
views
Does Amazon RedShift replace Hadoop for ~1XTB data?
There is plenty of hype surrounding Hadoop and its eco-system. However, in practice, where many data sets are in the terabyte range, is it not more reasonable to use Amazon RedShift for querying ...
- 253
11
votes
3
answers
711
views
Can map-reduce algorithms written for MongoDB be ported to Hadoop later?
In our company, we have a MongoDB database containing a lot of unstructured data, on which we need to run map-reduce algorithms to generate reports and other analyses. We have two approaches to select ...
- 1,393