Questions tagged [apache-hadoop]
Hadoop is an Apache open-source project that provides software for reliable and scalable distributed computing. The project itself includes a variety of other complementary additions.
23
questions with no upvoted or accepted answers
2
votes
0
answers
252
views
How to create tensors in spark?
I have the following data stored in HDFS: each row has three columns, id, date, item, which means a person with a particular id bought a particular item on a particular date. The dataset has billions ...
- 121
2
votes
0
answers
88
views
Why does my master node get heap memory full for inbuilt SVD API in Apache Spark during calculation of inverse of a square matrix?
I am using a three node system:
Master node: $64$ GB RAM
2 slave node: $32$ GB RAM
I am calculating the inverse of a square matrix using inbuilt Apache Spark function SVD. When I called SVD function ...
- 301
2
votes
0
answers
1k
views
How append works in hdfs? Where the newly created instance of file is placed?
I'm a Newbie to Hadoop!
By web search and after going through hadoop guides, it is clear that hdfs doesn't allow us to edit the file but to append some data to existing file, it will be creating a ...
- 21
1
vote
0
answers
18
views
what is a data lake catalog technically?
Is a datalake catalog, a file (XML) that the datalake consults before storing retrieving data from a Hadoop database?
Thanks.
- 17
1
vote
0
answers
112
views
Word count with map reduce
Suppose we use an input file that contains the following lyrics from a famous song:
We’re up all night till the sun
We’re up all night to get some
The input pairs for the Map phase will be the ...
- 151
1
vote
0
answers
39
views
Loading file into and out of HDFS via system call/cmd line vs using libhdfs
I am trying to implement a simple C/C++ program for the HDFS file system like word count, it takes a file from the input path puts it into HDFS (where it gets split), processed my map-reduce function ...
- 11
1
vote
0
answers
25
views
Processing XML via StreamXmlRecordReader and Pig in parallel
I'm trying to process a single large XML file with many records and then parse each record in parallel using AWS EMR.
...
- 111
1
vote
0
answers
123
views
How to push tasks to remote Hadoop/Spark cluster
I have access to a Hadoop cluster provided by a third party. Currently, I use Spark on this cluster by logging in to a Zeppelin ...
- 7,366
1
vote
0
answers
682
views
Datanode not starting on Slave Nodes for Apache Hadoop+Spark Setup
I know similar questions have been asked and I already looked through a lot of them to figure out what the issue might be, but to no avail.
I am a College student currently working on a Big Data ...
- 11
1
vote
0
answers
126
views
Unable to open application master UI in spark1.6.1 in cluster mode
Hadoop cluster has 8 nodes with high availability of resource manager. Active ResourceManager is in node 3. and Standby ResourceManager in node 2.
when i submit the application in cluster mode. the ...
- 11
1
vote
1
answer
318
views
Change log level of yarn?
I'd like to change the log level of yarn. For example, when I run yarn node -list, I see a line that says ...
- 601
1
vote
0
answers
108
views
Tasks of a Yarn Process in Hadoop
I am new to big data and Apache Hadoop YARN. This picture show how Yarn runs applications:
A client requests a Yarn Application. So, in a certain datanode a Container is created and the process is ...
- 161
1
vote
1
answer
36
views
Hdfs Data Balance on Cluster
Im new to Big Data, so please just dont blast me.
Can you explain me why HDFS works
best when the file blocks are evenly spread across the cluster ?
- 161
1
vote
0
answers
175
views
How to multiply a "fat and short" matrix with a "tall and thin" matrix using MapReduce?
Assume that $A_{m \times n}$ and $B_{n \times k}$ are to be multiplied to get $C_{m \times k}$. Now if $n$ is too large for a single row $A_{j}$ of $A$ to fit in RAM (and similarly for columns of B) ...
- 1,297
1
vote
0
answers
118
views
freebcp getting stalled for huge data
I am working on a project which processes big data (size ~3TB) every day. The first stage of our data pipeline copies the data from our MS-SQL Server to a host machine(linux server) using a tool ...
- 111
1
vote
0
answers
259
views
Pig is not able to read the complete data
I am trying to load a huge dataset of around 3.4 TB with approximately 1.4 million files in Pig on Amazon EMR.
The operations on the data are simple (JOIN and STORE), but the data is not getting ...
- 373
1
vote
0
answers
570
views
How to use REST API to execute Map-Reduce Task?
Let's say I have one Text File (Size 1 GB). I want to search particular word from file and if it is found the Line number should be returned.
I can Execute my java program using command line in linux....
- 131
1
vote
0
answers
216
views
Hadoop/Pig Aggregate Data
I am working on a project with two data sets. A time vs. speed data set (let's call it traffic), and a time vs. weather data set (called weather).
I am looking to find a correlation between these ...
- 91
0
votes
1
answer
43
views
Hadoop, Spark and Cloud
It seems Hadoop, Spark, and different versions of Clouds offer facilities to store and analyze big data. There are some articles comparing Hadoop and Spark (for example, this article). There are also ...
- 1
0
votes
1
answer
202
views
Storage of N-dimensional matrices (tensors) as part of machine learning pipelines
I'm an infra person working on a storage product. I've been googling quite a bit to find an answer to the following question but unable to do so. Hence, I am attemping to ask the question here.
I am ...
- 101
0
votes
1
answer
31
views
Can Single Node Hadoop Cluster be installed on a system with 1gb RAM
I am trying to learn hadoop, would like to know if for basic single node installation 1gb RAM system would be enough or we need more RAM. It would be helpful if someone can share what other minimum ...
- 101
0
votes
1
answer
23
views
What are common problems around HADOOP storage?
I've been asked to lead a program to understand why our Hadoop storage is constantly near capacity. What questions should I ask?
Data age,
Data size?
Housekeeping schedule?
How do we identify the ...
- 101
-1
votes
1
answer
93
views
Connecting Twitter API to a Big Data Environment?
I am trying to get live twitter data and store it in a Hadoop environment. Currently, I am learning how to push the data into a Cloudera environment and process it.
My questions are:
Is Cloudera ...