Questions tagged [apache-hadoop]
Hadoop is an Apache open-source project that provides software for reliable and scalable distributed computing. The project itself includes a variety of other complementary additions.
108
questions
0
votes
0answers
15 views
how to run pyspark on Hadoop cluster, cache data, then read file using pandas on local machine via its URI?
I connect to a Hadoop cluster via vpn. I want to execute a script that makes sql call and saves the data to a file on the remote machine, then using its URI I want to read on my local running Jupiter ...
1
vote
0answers
11 views
How to create tensors in spark?
I have the following data stored in HDFS: each row has three columns, id, date, item, which means a person with a particular id bought a particular item on a particular date. The dataset has billions ...
1
vote
0answers
21 views
Mapreduce jobs not working in hive
I was trying to execute a query
select name, count(*)
from amazon
where review != NULL
group by name ;
Number of reduce tasks not specified. Estimated ...
0
votes
0answers
21 views
Confused about testing graph
I am doing an image classification project using CNN (convolution neural network) model. I have plotted a training and validation graph. I was asked to plot a testing graph. I don’t know about testing ...
0
votes
1answer
26 views
What is the MapReduce application master?
From Hadoop The Definitive Guide
The whole process is illustrated in Figure 7-1. At the highest level,
there are five independent entities:
• The client, which submits the MapReduce ...
0
votes
0answers
7 views
is there a step where replicated data across different Hadoop nodes “synchronize” when job done
assume there are 3 Hadoop nodes have the same 50MB file block and doing a word count job.
is there a step where that 3 nodes does the job separately and "synchronize" the result when job done?
0
votes
1answer
14 views
what does “blocks” mean in the context of HDFS ecosystem? is it the same concept in the context of a single hard disk drive?
According to Hortonworks
HDFS has demonstrated production scalability of up to 200 PB of
storage and a single cluster of 4500 servers, supporting close to a
billion files and blocks.
what does ...
0
votes
0answers
11 views
Hints on my EMR Spark Cluster
I am new to Spark, in particular I am tying to find out which are the best recommended properties to set to work on a 1 master 2 slaves cluster with 10GB memory each.
Do you know which are a good set ...
0
votes
0answers
178 views
Datanode not starting on Slave Nodes for Apache Hadoop+Spark Setup
I know similar questions have been asked and I already looked through a lot of them to figure out what the issue might be, but to no avail.
I am a College student currently working on a Big Data ...
0
votes
1answer
41 views
SAP HANA or Hadoop? [closed]
this is a question regarding a career choice. I am a fresher and I recently joined an MNC in Data Engineering team. There I was offered training in either Hadoop or SAP HANA. I am in doubt as to which ...
2
votes
1answer
130 views
Is there any point in learning Hadoop in 2018?
I'm a data analyst/scientist working mostly with the Python open source stack i.e. Pandas, scikit-learn, matplotlib, seaborn etc. I want to expand my toolbox and learn a distributed computing ...
0
votes
1answer
703 views
Accumulators in Spark (PySpark) without global variables?
BACKGROUND
Consider the following textbook example which uses accumulators to add vectors.
...
0
votes
1answer
2k views
PySpark: java.io.EOFException
System:
1 name node, 4 cores, 16 GB RAM
1 master node, 4 cores, 16 GB RAM
6 data nodes, 4 cores, 16 GB RAM each
6 worker nodes, 4 cores, 16 GB RAM each
around 5 Terabytes of storage space
The data ...
1
vote
0answers
17 views
Remote sensing image data storage in hadoop hdfs [closed]
I am new to hadoop environment. anyone kindly please help me how to store the remote sensing image data in hadoop hdfs? thanks in advance
1
vote
0answers
50 views
Unable to open application master UI in spark1.6.1 in cluster mode
Hadoop cluster has 8 nodes with high availability of resource manager. Active ResourceManager is in node 3. and Standby ResourceManager in node 2.
when i submit the application in cluster mode. the ...
3
votes
0answers
367 views
Communication between name node, data node and client in hadoop by analysing Packet Capturing
if I say that in hadoop cluster system... A client has 10GB data to send to name node. After analysing the "packet capturing" actually the name node doesn't divide the data for data node, ...
1
vote
0answers
77 views
Change log level of yarn?
I'd like to change the log level of yarn. For example, when I run yarn node -list, I see a line that says ...
0
votes
2answers
119 views
Are Hadoop and Python SciPy used for the same? [closed]
Right now
I've been doing a bit of research because I'm quite new to Big Data world. Among several other tools or frameworks, I've read about Apache Hadoop and Python for data analysis.
Specifically,...
2
votes
1answer
40 views
does storing file in hdfs parallelize it for Spark?
For Spark's RDD operations, data must be in shape of RDD or be parallelized using:
ParallelizedData = sc.parallelize(data)
My question is that if I store data in ...
2
votes
1answer
287 views
Best practice for developing using Spark [closed]
I am looking for any tips and best practice on how to develop applications using Spark. I currently have access to a cluster, with data as well as a version of Spark 2.1.0 on an edge node and IntelliJ ...
5
votes
1answer
929 views
Saving Large Spark ML Pipeline to HDFS
I'm having trouble saving a large (relative to spark.rpc.message.maxSize) Spark ML pipeline to HDFS. Specifically, when I try to save the model to HDFS, it gives me an error related to spark's maximum ...
0
votes
1answer
51 views
Which one of these tasks will benefit the most from SPARK?
My company processes Data (I am only an intern). We primarily use Hadoop. We're starting to deploy spark in production. Currently we have have two jobs, we will choose just one to begin with spark. ...
1
vote
0answers
84 views
Tasks of a Yarn Process in Hadoop
I am new to big data and Apache Hadoop YARN. This picture show how Yarn runs applications:
A client requests a Yarn Application. So, in a certain datanode a Container is created and the process is ...
1
vote
0answers
31 views
Hdfs Data Balance on Cluster
Im new to Big Data, so please just dont blast me.
Can you explain me why HDFS works
best when the file blocks are evenly spread across the cluster ?
1
vote
0answers
270 views
Hadoop - checksum while reading file from client [closed]
In hadoop , if i want to read a file from client , the class DFSInputStream enters in action.
About this, my textbook says
The DFSInputStream also verifies checksums for the data transferred to it ...
1
vote
0answers
296 views
Custom Writable Serialization in Hadoop [closed]
I have some questions about Custom Writable objects in Hadoop. First of all, the private instance variables of the Writable object have to be Writable?
How does Hadoop handle ...
2
votes
1answer
19 views
Yarn service parameter for pseudodistributed [closed]
I'm trying to set up Hadoop in the pseudodistributed form. So I edited the suggested xml files. In the file yarn-site.xml, I had to insert:
...
1
vote
1answer
44 views
Hadoop and input informations divided in splits [closed]
Hadoop divides the input to a MapReduce job into fixed-size pieces
called input splits, or just splits. Hadoop creates one map task for
each split, which runs the user-defined map function for ...
2
votes
0answers
59 views
Why does my master node get heap memory full for inbuilt SVD API in Apache Spark during calculation of inverse of a square matrix?
I am using a three node system:
Master node: $64$ GB RAM
2 slave node: $32$ GB RAM
I am calculating the inverse of a square matrix using inbuilt Apache Spark function SVD. When I called SVD function ...
0
votes
1answer
92 views
Install Spark and Hadoop in the same machine [closed]
I'm trying to install Spark Apache and Hadoop in the same machine. Spark will be used to process data, and HDFS from Hadoop will be used to store data. I first started by installing spark and it ...
1
vote
0answers
277 views
Hadoop Cluster Capacity Planning [closed]
I was doing some digging to get some deeper understanding on the Capacity Planning done for setting up a Hadoop Cluster.
I have a daily ~100 GB of data generated and would like to find how a Capacity ...
3
votes
0answers
210 views
How to setup a home-laptop cluster to 'practice' elasticsearch, hadoop, mesos and spark [closed]
I'd like to have elasticsearch, hadoop, mesos and spark on my home laptop (it's an 8GB with an intel i7 CPU) for practicing the setup of such frameworks and eventually to test their functions.
I'm ...
2
votes
2answers
564 views
Why has Hadoop failed to become popular?
Apache Hadoop was once considered one of the tools you should have as a Data Scientist. Around 2012 to 2014, it gained the popularity steeply among Data Scientists, and seemed to be considered one of ...
2
votes
0answers
73 views
Are jobs the only way out for data scientists? [closed]
I have reached this point in my data sceince journey that I am left wondering if jobs are the only way out for me...
I am highly skilled in Hadoop, Machine Learning and now getting into Deep Learning ...
1
vote
1answer
72 views
what ETL technique should i use for text documents using Hadoop?
I have a school Big Data project where basically the teacher is going to give us a large amount of text documents (from the Gutenberg project data set ) and he want us to give as output the document ...
1
vote
2answers
330 views
Can R + Hadoop overcome R's memory constraints in any case?
I am an R user and I am interested in learning/understanding how Hadoop actually works. For this I previously read about Hadoop but was not able to find a satisfactory answer for my question.
Can R + ...
3
votes
1answer
126 views
Deploying models on bigdata platforms like Hadoop and Spark
Where exactly bigdata platforms fit in to a data science/ machine learning projects ?
Say I have a large dataset for a binary classification problem - cats and dogs.
Now I need to create a model for ...
2
votes
1answer
853 views
Ingestion of periodic REST API Calls into Hadoop [closed]
We are planning to use REST API calls to ingest data from an endpoint and store the data to HDFS. The REST calls are done in a periodic fashion (daily or maybe hourly).
I've already done Twitter ...
2
votes
1answer
179 views
K-means clustering on big data stored on multiple nodes on HDFS
I have a very basic question about how the k-means clustering algorithm works when the data is stored in HDFS.
Does the training take place on the subset of data on each node for a fixed value of K( ...
2
votes
2answers
108 views
How many people can use a single Hadoop cluster at one time?
I would like to know how many people can use a single Hadoop cluster at one time? I am asking because I need to figure out whether or not a single 5 or 10 node cluster would be sufficient to host a ...
-5
votes
1answer
196 views
Does a big data virtual machine machine help in analyzing large file?
I have a csv file of size 2.5GB. I can not analyze the data of the file in chunks. I don’t have access to a hadoop cluster. But, does running a virtual machine like hortonworks solve my problem? Is ...
2
votes
1answer
288 views
Mahout Spark shell not working
I installed Hadoop, Mahout and Spark. I am able to see the Hadoop and Spark MasterWebUI. Moreover, I can also run the following command,
...
2
votes
2answers
4k views
spark item similarity recommendation
I would like to build an recommendation engine using spark's Mlib itemsimilarity as mentioned here LINK
But it seems spark do not have this algorithm any more and ...
1
vote
1answer
174 views
How to Scaling Out Artifical Neural Networks?
I have started to study ANNs with Tensorflow and Keras. Now I want to find a solution to use ANNs over Hadoop. I have learnt that Spark 2.0 does have a Multilayer Perceptron Classifier, but as far as ...
-1
votes
1answer
84 views
Connecting Twitter API to a Big Data Environment?
I am trying to get live twitter data and store it in a Hadoop environment. Currently, I am learning how to push the data into a Cloudera environment and process it.
My questions are:
Is Cloudera ...
0
votes
1answer
736 views
Machine Learning model to find items that are frequently bought together using Hadoop Spark
I need to do my master thesis under Big Data Analytics - Machine Learning subject. I was challenged to create a Market Basket Analysis project on retail industry.
My dataset contains transactions of ...
0
votes
1answer
221 views
Spark algorithm to make a link analysis
I'm doing my master thesis on Big Data Analytics. I'm trying to develop a algorithm to identify associations between some products in a supermakert. Imagine that I've this dataset:
...
4
votes
8answers
555 views
Skills that school doesn't teach you
I'll be graduating soon with a master's degree in electrical engineering. I have a lot of experience with the theoretical side of machine learning through courses and research, but all of my ...
0
votes
3answers
266 views
Predictive Analytics on distributed systems vs standalone system
I have somewhat philosophical question regarding performing predictive analytics on distributed systems (such as hadoop). I am no expert on this subject so maybe other folks who have more advance ...
1
vote
0answers
107 views
How to multiply a “fat and short” matrix with a “tall and thin” matrix using MapReduce?
Assume that $A_{m \times n}$ and $B_{n \times k}$ are to be multiplied to get $C_{m \times k}$. Now if $n$ is too large for a single row $A_{j}$ of $A$ to fit in RAM (and similarly for columns of B) ...
