Questions tagged [apache-hadoop]

Hadoop is an Apache open-source project that provides software for reliable and scalable distributed computing. The project itself includes a variety of other complementary additions.

Filter by
Sorted by
Tagged with
0 votes
1 answer
37 views

Hadoop, Spark and Cloud

It seems Hadoop, Spark, and different versions of Clouds offer facilities to store and analyze big data. There are some articles comparing Hadoop and Spark (for example, this article). There are also ...
Tara's user avatar
  • 1
1 vote
0 answers
14 views

what is a data lake catalog technically?

Is a datalake catalog, a file (XML) that the datalake consults before storing retrieving data from a Hadoop database? Thanks.
SSSOF's user avatar
  • 13
0 votes
1 answer
31 views

Can I update the source of Data found in a Data Lake or Data Blob

Is it possible to update the source of data found in a Data Lake or Data Blob? What about while using HDInsight or Azure Databricks?
JF0001's user avatar
  • 101
0 votes
0 answers
4 views

How to custom mark a newly created Hadoop file?

I want to label the newly created Hadoop backups as unverified. I have come up with three ways: Name change to file.bak.unverified Name change to ...
Aviral Srivastava's user avatar
0 votes
0 answers
73 views

Hadoop creates two input splits (hence two maps) for a very small file : 14 bytes

I am working on a single node cluster, and i am running a wordcount job of a very small file, the file contains roughly 10 words, but the job creates two input splits which means two map jobs, i need ...
math geek's user avatar
0 votes
1 answer
30 views

What is the difference between Data Modeling and Data Processing?

When discussing big data, it is sometimes mentioned that data modeling can be done by using a tool like map reduce, while data processing may be performed by apache spark. What is the difference ...
Karl 17302's user avatar
0 votes
1 answer
164 views

Storage of N-dimensional matrices (tensors) as part of machine learning pipelines

I'm an infra person working on a storage product. I've been googling quite a bit to find an answer to the following question but unable to do so. Hence, I am attemping to ask the question here. I am ...
user855's user avatar
  • 101
1 vote
0 answers
104 views

Word count with map reduce

Suppose we use an input file that contains the following lyrics from a famous song: We’re up all night till the sun We’re up all night to get some The input pairs for the Map phase will be the ...
def __init__'s user avatar
0 votes
1 answer
29 views

Can Single Node Hadoop Cluster be installed on a system with 1gb RAM

I am trying to learn hadoop, would like to know if for basic single node installation 1gb RAM system would be enough or we need more RAM. It would be helpful if someone can share what other minimum ...
Gaurav Parek's user avatar
0 votes
1 answer
195 views

How to run unmodified Python program on GPU servers with scheduled GPUs?

Say I have one server with 10 GPUs. I have a python program which detects available GPU and use all of them. I have a couple of users who will run python (Machine learning or data mining) programs and ...
Gqqnbig's user avatar
  • 103
3 votes
1 answer
455 views

What is the main difference between Hadoop and Spark? [closed]

I recently read the following about Hadoop vs. Spark: Insist upon in-memory columnar data querying. This was the killer-feature that let Apache Spark run in seconds the queries that would take Hadoop ...
Ironclad's user avatar
0 votes
1 answer
294 views

Hive / Impala best practice code structuring

Coming from a DWH-background I am used to putting subqueries almost everywhere in my queries. On a Hadoop project (with Hive version 1.1.0 on Cloudera), I noticed we can forego subqueries in some ...
Gerardsson's user avatar
0 votes
1 answer
23 views

What are common problems around HADOOP storage?

I've been asked to lead a program to understand why our Hadoop storage is constantly near capacity. What questions should I ask? Data age, Data size? Housekeeping schedule? How do we identify the ...
vwdewaal's user avatar
  • 101
1 vote
0 answers
1k views

cannot access hive from spark [closed]

I am trying to install a hadoop + spark + hive cluster. I am using hadoop 3.1.2, spark 2.4.5 (scala 2.11 prebuilt with user-provided hadoop) and hive 2.3.3 (also tried 3.1.2 with the exact same ...
user3044083's user avatar
1 vote
0 answers
38 views

Loading file into and out of HDFS via system call/cmd line vs using libhdfs

I am trying to implement a simple C/C++ program for the HDFS file system like word count, it takes a file from the input path puts it into HDFS (where it gets split), processed my map-reduce function ...
n0unc3's user avatar
  • 11
1 vote
0 answers
25 views

Processing XML via StreamXmlRecordReader and Pig in parallel

I'm trying to process a single large XML file with many records and then parse each record in parallel using AWS EMR. ...
DaveO's user avatar
  • 111
3 votes
1 answer
466 views

BERT in production

I've created a BERT model. What are the ways to do the deployment of this model? Is it possible to use it with Spark, Hadoop or Docker?
illuminato's user avatar
2 votes
1 answer
5k views

Are parquet files compressed? [closed]

Parquet File Format Hadoop. Parquet, an open-source file format for Hadoop. Parquet stores nested data structures in a flat columnar format. Compared to a traditional approach where data is stored in ...
Pluviophile's user avatar
  • 3,550
1 vote
0 answers
117 views

How to push tasks to remote Hadoop/Spark cluster

I have access to a Hadoop cluster provided by a third party. Currently, I use Spark on this cluster by logging in to a Zeppelin ...
Peter's user avatar
  • 7,297
2 votes
0 answers
242 views

How to create tensors in spark?

I have the following data stored in HDFS: each row has three columns, id, date, item, which means a person with a particular id bought a particular item on a particular date. The dataset has billions ...
neverevernever's user avatar
0 votes
2 answers
1k views

What is the MapReduce application master?

From Hadoop The Definitive Guide The whole process is illustrated in Figure 7-1. At the highest level, there are five independent entities: • The client, which submits the MapReduce ...
Tim's user avatar
  • 151
0 votes
1 answer
20 views

what does "blocks" mean in the context of HDFS ecosystem? is it the same concept in the context of a single hard disk drive?

According to Hortonworks HDFS has demonstrated production scalability of up to 200 PB of storage and a single cluster of 4500 servers, supporting close to a billion files and blocks. what does ...
JJJohn's user avatar
  • 614
1 vote
0 answers
663 views

Datanode not starting on Slave Nodes for Apache Hadoop+Spark Setup

I know similar questions have been asked and I already looked through a lot of them to figure out what the issue might be, but to no avail. I am a College student currently working on a Big Data ...
Vinit's user avatar
  • 11
0 votes
1 answer
60 views

SAP HANA or Hadoop? [closed]

this is a question regarding a career choice. I am a fresher and I recently joined an MNC in Data Engineering team. There I was offered training in either Hadoop or SAP HANA. I am in doubt as to which ...
AswinRajaram's user avatar
4 votes
1 answer
401 views

Is there any point in learning Hadoop in 2018?

I'm a data analyst/scientist working mostly with the Python open source stack i.e. Pandas, scikit-learn, matplotlib, seaborn etc. I want to expand my toolbox and learn a distributed computing ...
RodrikTheReader's user avatar
2 votes
2 answers
3k views

Accumulators in Spark (PySpark) without global variables?

BACKGROUND Consider the following textbook example which uses accumulators to add vectors. ...
Tfovid's user avatar
  • 203
2 votes
2 answers
8k views

PySpark: java.io.EOFException

System: 1 name node, 4 cores, 16 GB RAM 1 master node, 4 cores, 16 GB RAM 6 data nodes, 4 cores, 16 GB RAM each 6 worker nodes, 4 cores, 16 GB RAM each around 5 Terabytes of storage space The data ...
dustin's user avatar
  • 131
1 vote
0 answers
24 views

Remote sensing image data storage in hadoop hdfs [closed]

I am new to hadoop environment. anyone kindly please help me how to store the remote sensing image data in hadoop hdfs? thanks in advance
Harun Ijaz's user avatar
1 vote
0 answers
123 views

Unable to open application master UI in spark1.6.1 in cluster mode

Hadoop cluster has 8 nodes with high availability of resource manager. Active ResourceManager is in node 3. and Standby ResourceManager in node 2. when i submit the application in cluster mode. the ...
shashikumar's user avatar
3 votes
0 answers
571 views

Communication between name node, data node and client in hadoop by analysing Packet Capturing [closed]

if I say that in hadoop cluster system... A client has 10GB data to send to name node. After analysing the "packet capturing" actually the name node doesn't divide the data for data node, ...
Radheyshyam Sain's user avatar
1 vote
1 answer
297 views

Change log level of yarn?

I'd like to change the log level of yarn. For example, when I run yarn node -list, I see a line that says ...
vy32's user avatar
  • 601
0 votes
2 answers
206 views

Are Hadoop and Python SciPy used for the same? [closed]

Right now I've been doing a bit of research because I'm quite new to Big Data world. Among several other tools or frameworks, I've read about Apache Hadoop and Python for data analysis. Specifically,...
Btc Sources's user avatar
2 votes
1 answer
68 views

does storing file in hdfs parallelize it for Spark?

For Spark's RDD operations, data must be in shape of RDD or be parallelized using: ParallelizedData = sc.parallelize(data) My question is that if I store data in ...
Ali Majed HA's user avatar
2 votes
1 answer
385 views

Best practice for developing using Spark [closed]

I am looking for any tips and best practice on how to develop applications using Spark. I currently have access to a cluster, with data as well as a version of Spark 2.1.0 on an edge node and IntelliJ ...
Taylrl's user avatar
  • 488
5 votes
1 answer
2k views

Saving Large Spark ML Pipeline to HDFS

I'm having trouble saving a large (relative to spark.rpc.message.maxSize) Spark ML pipeline to HDFS. Specifically, when I try to save the model to HDFS, it gives me an error related to spark's maximum ...
Thomas Cleberg's user avatar
0 votes
1 answer
69 views

Which one of these tasks will benefit the most from SPARK?

My company processes Data (I am only an intern). We primarily use Hadoop. We're starting to deploy spark in production. Currently we have have two jobs, we will choose just one to begin with spark. ...
Melchia's user avatar
  • 111
1 vote
0 answers
107 views

Tasks of a Yarn Process in Hadoop

I am new to big data and Apache Hadoop YARN. This picture show how Yarn runs applications: A client requests a Yarn Application. So, in a certain datanode a Container is created and the process is ...
Tantaros's user avatar
  • 161
1 vote
1 answer
36 views

Hdfs Data Balance on Cluster

Im new to Big Data, so please just dont blast me. Can you explain me why HDFS works best when the file blocks are evenly spread across the cluster ?
Tantaros's user avatar
  • 161
1 vote
0 answers
300 views

Hadoop - checksum while reading file from client [closed]

In hadoop , if i want to read a file from client , the class DFSInputStream enters in action. About this, my textbook says The DFSInputStream also verifies checksums for the data transferred to it ...
Tantaros's user avatar
  • 161
1 vote
0 answers
381 views

Custom Writable Serialization in Hadoop [closed]

I have some questions about Custom Writable objects in Hadoop. First of all, the private instance variables of the Writable object have to be Writable? How does Hadoop handle ...
VanBaffo's user avatar
  • 111
2 votes
1 answer
23 views

Yarn service parameter for pseudodistributed [closed]

I'm trying to set up Hadoop in the pseudodistributed form. So I edited the suggested xml files. In the file yarn-site.xml, I had to insert: ...
Poiera's user avatar
  • 451
1 vote
1 answer
49 views

Hadoop and input informations divided in splits [closed]

Hadoop divides the input to a MapReduce job into fixed-size pieces called input splits, or just splits. Hadoop creates one map task for each split, which runs the user-defined map function for ...
Poiera's user avatar
  • 451
2 votes
0 answers
88 views

Why does my master node get heap memory full for inbuilt SVD API in Apache Spark during calculation of inverse of a square matrix?

I am using a three node system: Master node: $64$ GB RAM 2 slave node: $32$ GB RAM I am calculating the inverse of a square matrix using inbuilt Apache Spark function SVD. When I called SVD function ...
Chandan Gautam's user avatar
0 votes
1 answer
139 views

Install Spark and Hadoop in the same machine [closed]

I'm trying to install Spark Apache and Hadoop in the same machine. Spark will be used to process data, and HDFS from Hadoop will be used to store data. I first started by installing spark and it ...
Mehdi Ben Hamida's user avatar
1 vote
0 answers
296 views

Hadoop Cluster Capacity Planning [closed]

I was doing some digging to get some deeper understanding on the Capacity Planning done for setting up a Hadoop Cluster. I have a daily ~100 GB of data generated and would like to find how a Capacity ...
DataWrangler's user avatar
3 votes
0 answers
227 views

How to setup a home-laptop cluster to 'practice' elasticsearch, hadoop, mesos and spark [closed]

I'd like to have elasticsearch, hadoop, mesos and spark on my home laptop (it's an 8GB with an intel i7 CPU) for practicing the setup of such frameworks and eventually to test their functions. I'm ...
user1403546's user avatar
2 votes
2 answers
662 views

Why has Hadoop failed to become popular?

Apache Hadoop was once considered one of the tools you should have as a Data Scientist. Around 2012 to 2014, it gained the popularity steeply among Data Scientists, and seemed to be considered one of ...
Blaszard's user avatar
  • 901
2 votes
0 answers
79 views

Are jobs the only way out for data scientists? [closed]

I have reached this point in my data sceince journey that I am left wondering if jobs are the only way out for me... I am highly skilled in Hadoop, Machine Learning and now getting into Deep Learning ...
Sudi's user avatar
  • 21
1 vote
1 answer
159 views

what ETL technique should i use for text documents using Hadoop?

I have a school Big Data project where basically the teacher is going to give us a large amount of text documents (from the Gutenberg project data set ) and he want us to give as output the document ...
Sebastian Delgado's user avatar
1 vote
2 answers
390 views

Can R + Hadoop overcome R's memory constraints in any case?

I am an R user and I am interested in learning/understanding how Hadoop actually works. For this I previously read about Hadoop but was not able to find a satisfactory answer for my question. Can R + ...
Alex's user avatar
  • 113