Questions tagged [apache-hadoop]
Hadoop is an Apache open-source project that provides software for reliable and scalable distributed computing. The project itself includes a variety of other complementary additions.
118
questions
0
votes
1
answer
123
views
Hadoop, Spark and Cloud
It seems Hadoop, Spark, and different versions of Clouds offer facilities to store and analyze big data. There are some articles comparing Hadoop and Spark (for example, this article). There are also ...
CommunityBot
- 1
0
votes
1
answer
386
views
Storage of N-dimensional matrices (tensors) as part of machine learning pipelines
I'm an infra person working on a storage product. I've been googling quite a bit to find an answer to the following question but unable to do so. Hence, I am attemping to ask the question here.
I am ...
CommunityBot
- 1
1
vote
1
answer
355
views
Change log level of yarn?
I'd like to change the log level of yarn. For example, when I run yarn node -list, I see a line that says ...
CommunityBot
- 1
1
vote
2
answers
6k
views
how to disable query from beeline results
I am occuring a strange hive-client beeline behavior.
In the outputed file with query results there is also a queary at the ...
- 1
1
vote
0
answers
18
views
what is a data lake catalog technically?
Is a datalake catalog, a file (XML) that the datalake consults before storing retrieving data from a Hadoop database?
Thanks.
- 17
0
votes
1
answer
38
views
Can I update the source of Data found in a Data Lake or Data Blob
Is it possible to update the source of data found in a Data Lake or Data Blob? What about while using HDInsight or Azure Databricks?
- 306
1
vote
1
answer
38
views
What is the difference between Data Modeling and Data Processing?
When discussing big data, it is sometimes mentioned that data modeling can be done by using a tool like map reduce, while data processing may be performed by apache spark. What is the difference ...
- 25.7k
40
votes
10
answers
21k
views
Do I need to learn Hadoop to be a Data Scientist?
An aspiring data scientist here. I don't know anything about Hadoop, but as I have been reading about Data Science and Big Data, I see a lot of talk about Hadoop. Is it absolutely necessary to learn ...
- 411
1
vote
1
answer
40
views
Hdfs Data Balance on Cluster
Im new to Big Data, so please just dont blast me.
Can you explain me why HDFS works
best when the file blocks are evenly spread across the cluster ?
- 411
6
votes
3
answers
2k
views
Is there a benefit to using hadoop with only one node?
I just started learning about Hadoop. From what I understand its primary strength is its ability to distribute a task across many nodes. Is there any benefit to using Hadoop with only a single node, ...
- 411
2
votes
2
answers
3k
views
Accumulators in Spark (PySpark) without global variables?
BACKGROUND
Consider the following textbook example which uses accumulators to add vectors.
...
CommunityBot
- 1
1
vote
0
answers
117
views
Word count with map reduce
Suppose we use an input file that contains the following lyrics from a famous song:
We’re up all night till the sun
We’re up all night to get some
The input pairs for the Map phase will be the ...
- 151
2
votes
1
answer
71
views
does storing file in hdfs parallelize it for Spark?
For Spark's RDD operations, data must be in shape of RDD or be parallelized using:
ParallelizedData = sc.parallelize(data)
My question is that if I store data in ...
- 1,012
5
votes
1
answer
2k
views
Saving Large Spark ML Pipeline to HDFS
I'm having trouble saving a large (relative to spark.rpc.message.maxSize) Spark ML pipeline to HDFS. Specifically, when I try to save the model to HDFS, it gives me an error related to spark's maximum ...
- 1,012
2
votes
2
answers
10k
views
PySpark: java.io.EOFException
System:
1 name node, 4 cores, 16 GB RAM
1 master node, 4 cores, 16 GB RAM
6 data nodes, 4 cores, 16 GB RAM each
6 worker nodes, 4 cores, 16 GB RAM each
around 5 Terabytes of storage space
The data ...
- 11
0
votes
1
answer
32
views
Can Single Node Hadoop Cluster be installed on a system with 1gb RAM
I am trying to learn hadoop, would like to know if for basic single node installation 1gb RAM system would be enough or we need more RAM. It would be helpful if someone can share what other minimum ...
CommunityBot
- 1
0
votes
1
answer
60
views
SAP HANA or Hadoop? [closed]
this is a question regarding a career choice. I am a fresher and I recently joined an MNC in Data Engineering team. There I was offered training in either Hadoop or SAP HANA. I am in doubt as to which ...
- 6,422
0
votes
1
answer
208
views
How to run unmodified Python program on GPU servers with scheduled GPUs?
Say I have one server with 10 GPUs. I have a python program which detects available GPU and use all of them.
I have a couple of users who will run python (Machine learning or data mining) programs and ...
- 27.2k
0
votes
1
answer
23
views
What are common problems around HADOOP storage?
I've been asked to lead a program to understand why our Hadoop storage is constantly near capacity. What questions should I ask?
Data age,
Data size?
Housekeeping schedule?
How do we identify the ...
CommunityBot
- 1
4
votes
1
answer
952
views
What is the main difference between Hadoop and Spark? [closed]
I recently read the following about Hadoop vs. Spark:
Insist upon in-memory columnar data querying. This was the killer-feature that let Apache Spark run in seconds the queries that would take Hadoop ...
- 661
0
votes
2
answers
1k
views
What is the MapReduce application master?
From Hadoop The Definitive Guide
The whole process is illustrated in Figure 7-1. At the highest level,
there are five independent entities:
• The client, which submits the MapReduce ...
- 262
6
votes
1
answer
3k
views
How to make k-means distributed?
After setting up a 2-noded Hadoop cluster, understanding Hadoop and Python and based on this naive implementation, I ended up with this code:
...
- 997
0
votes
1
answer
471
views
Hive / Impala best practice code structuring
Coming from a DWH-background I am used to putting subqueries almost everywhere in my queries. On a Hadoop project (with Hive version 1.1.0 on Cloudera), I noticed we can forego subqueries in some ...
CommunityBot
- 1
34
votes
5
answers
13k
views
What are the use cases for Apache Spark vs Hadoop
With Hadoop 2.0 and YARN Hadoop is supposedly no longer tied only map-reduce solutions. With that advancement, what are the use cases for Apache Spark vs Hadoop considering both sit atop of HDFS? I've ...
- 452
3
votes
1
answer
570
views
BERT in production
I've created a BERT model. What are the ways to do the deployment of this model? Is it possible to use it with Spark, Hadoop or Docker?
CommunityBot
- 1
1
vote
0
answers
1k
views
cannot access hive from spark [closed]
I am trying to install a hadoop + spark + hive cluster.
I am using hadoop 3.1.2, spark 2.4.5 (scala 2.11 prebuilt with user-provided hadoop) and hive 2.3.3 (also tried 3.1.2 with the exact same ...
- 11
1
vote
0
answers
40
views
Loading file into and out of HDFS via system call/cmd line vs using libhdfs
I am trying to implement a simple C/C++ program for the HDFS file system like word count, it takes a file from the input path puts it into HDFS (where it gets split), processed my map-reduce function ...
- 11
1
vote
0
answers
25
views
Processing XML via StreamXmlRecordReader and Pig in parallel
I'm trying to process a single large XML file with many records and then parse each record in parallel using AWS EMR.
...
- 111
3
votes
1
answer
6k
views
Are parquet files compressed? [closed]
Parquet File Format Hadoop. Parquet, an open-source file format for Hadoop. Parquet stores nested data structures in a flat columnar format. Compared to a traditional approach where data is stored in ...
- 5,739
1
vote
0
answers
128
views
How to push tasks to remote Hadoop/Spark cluster
I have access to a Hadoop cluster provided by a third party. Currently, I use Spark on this cluster by logging in to a Zeppelin ...
- 7,596
2
votes
0
answers
276
views
How to create tensors in spark?
I have the following data stored in HDFS: each row has three columns, id, date, item, which means a person with a particular id bought a particular item on a particular date. The dataset has billions ...
- 121
-1
votes
1
answer
95
views
Connecting Twitter API to a Big Data Environment?
I am trying to get live twitter data and store it in a Hadoop environment. Currently, I am learning how to push the data into a Cloudera environment and process it.
My questions are:
Is Cloudera ...
- 1,655
4
votes
2
answers
100
views
Pig script code error?
While running the below pig script I am getting an error in line4:
If it is GROUP then I am getting error.
If I change from 'GROUP' TO 'group' in line4, then the script is running.
What is the ...
- 1,655
0
votes
1
answer
21
views
what does "blocks" mean in the context of HDFS ecosystem? is it the same concept in the context of a single hard disk drive?
According to Hortonworks
HDFS has demonstrated production scalability of up to 200 PB of
storage and a single cluster of 4500 servers, supporting close to a
billion files and blocks.
what does ...
- 166
1
vote
0
answers
723
views
Datanode not starting on Slave Nodes for Apache Hadoop+Spark Setup
I know similar questions have been asked and I already looked through a lot of them to figure out what the issue might be, but to no avail.
I am a College student currently working on a Big Data ...
- 11
2
votes
0
answers
91
views
Why does my master node get heap memory full for inbuilt SVD API in Apache Spark during calculation of inverse of a square matrix?
I am using a three node system:
Master node: $64$ GB RAM
2 slave node: $32$ GB RAM
I am calculating the inverse of a square matrix using inbuilt Apache Spark function SVD. When I called SVD function ...
- 376
4
votes
1
answer
404
views
Is there any point in learning Hadoop in 2018?
I'm a data analyst/scientist working mostly with the Python open source stack i.e. Pandas, scikit-learn, matplotlib, seaborn etc. I want to expand my toolbox and learn a distributed computing ...
CommunityBot
- 1
1
vote
0
answers
28
views
Remote sensing image data storage in hadoop hdfs [closed]
I am new to hadoop environment. anyone kindly please help me how to store the remote sensing image data in hadoop hdfs? thanks in advance
- 11
1
vote
0
answers
135
views
Unable to open application master UI in spark1.6.1 in cluster mode
Hadoop cluster has 8 nodes with high availability of resource manager. Active ResourceManager is in node 3. and Standby ResourceManager in node 2.
when i submit the application in cluster mode. the ...
- 11
3
votes
0
answers
589
views
Communication between name node, data node and client in hadoop by analysing Packet Capturing [closed]
if I say that in hadoop cluster system... A client has 10GB data to send to name node. After analysing the "packet capturing" actually the name node doesn't divide the data for data node, ...
- 8,064
0
votes
2
answers
216
views
Are Hadoop and Python SciPy used for the same? [closed]
Right now
I've been doing a bit of research because I'm quite new to Big Data world. Among several other tools or frameworks, I've read about Apache Hadoop and Python for data analysis.
Specifically,...
- 2,440
3
votes
1
answer
263
views
Deploying models on bigdata platforms like Hadoop and Spark
Where exactly bigdata platforms fit in to a data science/ machine learning projects ?
Say I have a large dataset for a binary classification problem - cats and dogs.
Now I need to create a model for ...
- 405
2
votes
1
answer
2k
views
Error when using MAX in Apache Pig (Hadoop)
I am trying to calculate maximum values for different groups in a relation in Pig. The relation has three columns patientid, featureid and featurevalue (all int).
I group the relation based on ...
1
vote
1
answer
162
views
what ETL technique should i use for text documents using Hadoop?
I have a school Big Data project where basically the teacher is going to give us a large amount of text documents (from the Gutenberg project data set ) and he want us to give as output the document ...
2
votes
1
answer
393
views
Best practice for developing using Spark [closed]
I am looking for any tips and best practice on how to develop applications using Spark. I currently have access to a cluster, with data as well as a version of Spark 2.1.0 on an edge node and IntelliJ ...
- 1,742
0
votes
1
answer
70
views
Which one of these tasks will benefit the most from SPARK?
My company processes Data (I am only an intern). We primarily use Hadoop. We're starting to deploy spark in production. Currently we have have two jobs, we will choose just one to begin with spark. ...
- 3,940
1
vote
0
answers
112
views
Tasks of a Yarn Process in Hadoop
I am new to big data and Apache Hadoop YARN. This picture show how Yarn runs applications:
A client requests a Yarn Application. So, in a certain datanode a Container is created and the process is ...
- 1,783
1
vote
0
answers
301
views
Hadoop - checksum while reading file from client [closed]
In hadoop , if i want to read a file from client , the class DFSInputStream enters in action.
About this, my textbook says
The DFSInputStream also verifies checksums for the data transferred to it ...
- 161
1
vote
0
answers
387
views
Custom Writable Serialization in Hadoop [closed]
I have some questions about Custom Writable objects in Hadoop. First of all, the private instance variables of the Writable object have to be Writable?
How does Hadoop handle ...
- 1,783
2
votes
1
answer
24
views
Yarn service parameter for pseudodistributed [closed]
I'm trying to set up Hadoop in the pseudodistributed form. So I edited the suggested xml files. In the file yarn-site.xml, I had to insert:
...
- 1,783