Questions tagged [map-reduce]
MapReduce is a programming model for processing large data sets with a parallel, distributed algorithm on a cluster.
28 questions
1
vote
0
answers
550
views
Best Practice to handle large data in HDBSCAN cluserting using tfidf
I am now about to implement HDBSCAN on a very large set of data, a few million sentences, using tfidf. However, as we know ...
- 420
1
vote
1
answer
30
views
Would it be possible/practical to build a distributed deep learning engine by tapping into ordinary PCs' unused resources?
I started thinking about this in the context of Apple's new line of desktop CPUs with dedicated neural engines. From what I hear, these chips are quite adept at solving deep learning problems (as the ...
- 373
1
vote
0
answers
120
views
Word count with map reduce
Suppose we use an input file that contains the following lyrics from a famous song:
We’re up all night till the sun
We’re up all night to get some
The input pairs for the Map phase will be the ...
- 151
2
votes
1
answer
2k
views
Dataset map function error : TypeError: Expected list for 'input' argument to 'EagerPyFunc' Op, not Tensor
I am currently trying to write a script to create a TFRecord file.
Therefore, I am following the instruction on the offical tensorflow website: https://www.tensorflow.org/tutorials/load_data/tfrecord#...
- 171
1
vote
0
answers
42
views
Loading file into and out of HDFS via system call/cmd line vs using libhdfs
I am trying to implement a simple C/C++ program for the HDFS file system like word count, it takes a file from the input path puts it into HDFS (where it gets split), processed my map-reduce function ...
- 11
0
votes
2
answers
2k
views
What is the MapReduce application master?
From Hadoop The Definitive Guide
The whole process is illustrated in Figure 7-1. At the highest level,
there are five independent entities:
• The client, which submits the MapReduce ...
- 151
1
vote
0
answers
60
views
How to read in all text files from UNIX bash directory in Cloudera's Python API
I'm still pretty new to Cloudera and using the UNIX environment. I have written a mapper that reads in .txt files from a directory in my Windows system, which works just fine. I read files in like ...
- 111
0
votes
1
answer
44
views
How to generate list with out for loop in python [closed]
data frame
S/N Type Number Capacity
1 Bike 2 5
2 Tempo 1 30
3 Truck-1 1 60
4 Truck-2 1 90
I would like to generate capacitylist = [5,5,30,60,90]
Is it ...
0
votes
1
answer
73
views
Which one of these tasks will benefit the most from SPARK?
My company processes Data (I am only an intern). We primarily use Hadoop. We're starting to deploy spark in production. Currently we have have two jobs, we will choose just one to begin with spark. ...
- 111
1
vote
0
answers
179
views
How to multiply a "fat and short" matrix with a "tall and thin" matrix using MapReduce?
Assume that $A_{m \times n}$ and $B_{n \times k}$ are to be multiplied to get $C_{m \times k}$. Now if $n$ is too large for a single row $A_{j}$ of $A$ to fit in RAM (and similarly for columns of B) ...
- 1,297
1
vote
1
answer
116
views
Suggestions on what patterns/analysis to derive from Airlines Big Data [closed]
I recently started learning Hadoop,
I found this data set http://stat-computing.org/dataexpo/2009/the-data.html - (2009 data),
I want some suggestions as what type of patterns or analysis can I do in ...
- 191
6
votes
3
answers
4k
views
Why does map reduce have a shuffle step?
I'm looking at a diagram of map reduce where there is a map step, a shuffle step and then the reduce step. Why shuffle?
- 921
6
votes
1
answer
3k
views
How to make k-means distributed?
After setting up a 2-noded Hadoop cluster, understanding Hadoop and Python and based on this naive implementation, I ended up with this code:
...
- 291
1
vote
0
answers
265
views
Pig is not able to read the complete data
I am trying to load a huge dataset of around 3.4 TB with approximately 1.4 million files in Pig on Amazon EMR.
The operations on the data are simple (JOIN and STORE), but the data is not getting ...
- 373
0
votes
2
answers
365
views
Spark vs Map reduce
I know that Spark enhances performance relative to mapreduce by doing in-memory computations. But, since the caching is explicitly decided by the programmer, one can also proceed without doing that. I ...
- 154
2
votes
3
answers
273
views
Can Hadoop be beneficial when data is in database tables and not in a file system
I work for a bank. Most of our data is in the form of database tables. Would we benefit by implementing Hadoop? I am of the impression that Hadoop is more for a Distributed File System (unstructured ...
- 611
5
votes
4
answers
2k
views
Can all statistical algorithms be parallelized using a Map Reduce framework
Is it correct to say that any statistical learning algorithm (linear/logistic regression, SVM, neural network, random forest) can be implemented inside a Map Reduce framework? Or are there ...
- 611
4
votes
1
answer
86
views
Data produced as an output to Dumbo API of Python not getting distributed to all the nodes of cluster
On the node from which I run Dumbo commands, all the files produced as output are produced on the same node. For example, suppose there is a node having name hvs on which I ran the script:
dumbo ...
2
votes
1
answer
266
views
Time Complexity notation in Big Data platforms
I am redesigning some of the classical algorithms for Hadoop/MapReduce framework. I was wondering if there any established approach for denoting Big(O) kind of expressions to measure time complexity?
...
- 141
0
votes
1
answer
223
views
MPI, MapReduce, or Spark for complex datasets and processing
I have 2 data files: the first one is a database, potentially very large; the second one contains queries I want to answer. My program pipeline is processing the database to get some information first,...
- 101
4
votes
1
answer
45
views
Timing sequence in MapReduce
I'm running a test on MapReduce algorithm in different environments, like Hadoop and MongoDB, and using different types of data. What are the different methods or techniques to find out the execution ...
- 41
5
votes
3
answers
643
views
What technologies are fastest at performing joins on large datasets?
By "large", I mean in the range of 100m to 10b rows.
I'm currently using both Hadoop MapReduce and Amazon RedShift. MapReduce has been a little disappointing here. Redshift works very well if the ...
- 51
23
votes
3
answers
9k
views
Nearest neighbors search for very high dimensional data
I have a big sparse matrix of users and items they like (in the order of 1M users and 100K items, with a very low level of sparsity). I'm exploring ways in which I could perform kNN search on it. ...
- 451
8
votes
4
answers
1k
views
Data science and MapReduce programming model of Hadoop
What are the different classes of data science problems that can be solved using mapreduce programming model?
- 369
7
votes
1
answer
1k
views
Linear Regression in R Mapreduce(RHadoop)
I m new to RHadoop and also to RMR...
I had an requirement to write a Mapreduce Job in R Mapreduce. I have Tried writing but While executing this it gives an Error.
Tring to read the file from hdfs
...
- 101
2
votes
1
answer
865
views
Difference Between Hadoop Mapreduce(Java) and RHadoop mapreduce
I understand Hadoop MapReduce and its features but I am confused about R MapReduce.
One difference I have read is that R utilizes maximum RAM. So do perform parallel processing integrated R with ...
- 101
12
votes
3
answers
3k
views
Does Amazon RedShift replace Hadoop for ~1XTB data?
There is plenty of hype surrounding Hadoop and its eco-system. However, in practice, where many data sets are in the terabyte range, is it not more reasonable to use Amazon RedShift for querying ...
- 253
11
votes
3
answers
711
views
Can map-reduce algorithms written for MongoDB be ported to Hadoop later?
In our company, we have a MongoDB database containing a lot of unstructured data, on which we need to run map-reduce algorithms to generate reports and other analyses. We have two approaches to select ...
- 1,393