Questions tagged [apache-hadoop]

Hadoop is an Apache open-source project that provides software for reliable and scalable distributed computing. The project itself includes a variety of other complementary additions.

23 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
2 votes
0 answers
257 views

How to create tensors in spark?

I have the following data stored in HDFS: each row has three columns, id, date, item, which means a person with a particular id bought a particular item on a particular date. The dataset has billions ...
neverevernever's user avatar
2 votes
0 answers
90 views

Why does my master node get heap memory full for inbuilt SVD API in Apache Spark during calculation of inverse of a square matrix?

I am using a three node system: Master node: $64$ GB RAM 2 slave node: $32$ GB RAM I am calculating the inverse of a square matrix using inbuilt Apache Spark function SVD. When I called SVD function ...
Chandan Gautam's user avatar
2 votes
0 answers
1k views

How append works in hdfs? Where the newly created instance of file is placed?

I'm a Newbie to Hadoop! By web search and after going through hadoop guides, it is clear that hdfs doesn't allow us to edit the file but to append some data to existing file, it will be creating a ...
vtechie's user avatar
  • 21
1 vote
0 answers
18 views

what is a data lake catalog technically?

Is a datalake catalog, a file (XML) that the datalake consults before storing retrieving data from a Hadoop database? Thanks.
SSSOF's user avatar
  • 17
1 vote
0 answers
114 views

Word count with map reduce

Suppose we use an input file that contains the following lyrics from a famous song: We’re up all night till the sun We’re up all night to get some The input pairs for the Map phase will be the ...
def __init__'s user avatar
1 vote
0 answers
40 views

Loading file into and out of HDFS via system call/cmd line vs using libhdfs

I am trying to implement a simple C/C++ program for the HDFS file system like word count, it takes a file from the input path puts it into HDFS (where it gets split), processed my map-reduce function ...
n0unc3's user avatar
  • 11
1 vote
0 answers
25 views

Processing XML via StreamXmlRecordReader and Pig in parallel

I'm trying to process a single large XML file with many records and then parse each record in parallel using AWS EMR. ...
DaveO's user avatar
  • 111
1 vote
0 answers
127 views

How to push tasks to remote Hadoop/Spark cluster

I have access to a Hadoop cluster provided by a third party. Currently, I use Spark on this cluster by logging in to a Zeppelin ...
Peter's user avatar
  • 7,426
1 vote
0 answers
699 views

Datanode not starting on Slave Nodes for Apache Hadoop+Spark Setup

I know similar questions have been asked and I already looked through a lot of them to figure out what the issue might be, but to no avail. I am a College student currently working on a Big Data ...
Vinit's user avatar
  • 11
1 vote
0 answers
128 views

Unable to open application master UI in spark1.6.1 in cluster mode

Hadoop cluster has 8 nodes with high availability of resource manager. Active ResourceManager is in node 3. and Standby ResourceManager in node 2. when i submit the application in cluster mode. the ...
shashikumar's user avatar
1 vote
1 answer
326 views

Change log level of yarn?

I'd like to change the log level of yarn. For example, when I run yarn node -list, I see a line that says ...
vy32's user avatar
  • 601
1 vote
0 answers
110 views

Tasks of a Yarn Process in Hadoop

I am new to big data and Apache Hadoop YARN. This picture show how Yarn runs applications: A client requests a Yarn Application. So, in a certain datanode a Container is created and the process is ...
Tantaros's user avatar
  • 161
1 vote
1 answer
37 views

Hdfs Data Balance on Cluster

Im new to Big Data, so please just dont blast me. Can you explain me why HDFS works best when the file blocks are evenly spread across the cluster ?
Tantaros's user avatar
  • 161
1 vote
0 answers
176 views

How to multiply a "fat and short" matrix with a "tall and thin" matrix using MapReduce?

Assume that $A_{m \times n}$ and $B_{n \times k}$ are to be multiplied to get $C_{m \times k}$. Now if $n$ is too large for a single row $A_{j}$ of $A$ to fit in RAM (and similarly for columns of B) ...
wabbit's user avatar
  • 1,297
1 vote
0 answers
120 views

freebcp getting stalled for huge data

I am working on a project which processes big data (size ~3TB) every day. The first stage of our data pipeline copies the data from our MS-SQL Server to a host machine(linux server) using a tool ...
Ravi Chandra's user avatar
1 vote
0 answers
261 views

Pig is not able to read the complete data

I am trying to load a huge dataset of around 3.4 TB with approximately 1.4 million files in Pig on Amazon EMR. The operations on the data are simple (JOIN and STORE), but the data is not getting ...
shanky_thebearer's user avatar
1 vote
0 answers
571 views

How to use REST API to execute Map-Reduce Task?

Let's say I have one Text File (Size 1 GB). I want to search particular word from file and if it is found the Line number should be returned. I can Execute my java program using command line in linux....
spt025's user avatar
  • 131
1 vote
0 answers
216 views

Hadoop/Pig Aggregate Data

I am working on a project with two data sets. A time vs. speed data set (let's call it traffic), and a time vs. weather data set (called weather). I am looking to find a correlation between these ...
BigDataDude's user avatar
0 votes
1 answer
52 views

Hadoop, Spark and Cloud

It seems Hadoop, Spark, and different versions of Clouds offer facilities to store and analyze big data. There are some articles comparing Hadoop and Spark (for example, this article). There are also ...
Tara's user avatar
  • 1
0 votes
1 answer
238 views

Storage of N-dimensional matrices (tensors) as part of machine learning pipelines

I'm an infra person working on a storage product. I've been googling quite a bit to find an answer to the following question but unable to do so. Hence, I am attemping to ask the question here. I am ...
user855's user avatar
  • 101
0 votes
1 answer
31 views

Can Single Node Hadoop Cluster be installed on a system with 1gb RAM

I am trying to learn hadoop, would like to know if for basic single node installation 1gb RAM system would be enough or we need more RAM. It would be helpful if someone can share what other minimum ...
Gaurav Parek's user avatar
0 votes
1 answer
23 views

What are common problems around HADOOP storage?

I've been asked to lead a program to understand why our Hadoop storage is constantly near capacity. What questions should I ask? Data age, Data size? Housekeeping schedule? How do we identify the ...
vwdewaal's user avatar
  • 101
-1 votes
1 answer
94 views

Connecting Twitter API to a Big Data Environment?

I am trying to get live twitter data and store it in a Hadoop environment. Currently, I am learning how to push the data into a Cloudera environment and process it. My questions are: Is Cloudera ...
Melvin Roy's user avatar