Skip to main content

Questions tagged [apache-hadoop]

Hadoop is an Apache open-source project that provides software for reliable and scalable distributed computing. The project itself includes a variety of other complementary additions.

Filter by
Sorted by
Tagged with
40 votes
10 answers
21k views

Do I need to learn Hadoop to be a Data Scientist?

An aspiring data scientist here. I don't know anything about Hadoop, but as I have been reading about Data Science and Big Data, I see a lot of talk about Hadoop. Is it absolutely necessary to learn ...
Pensu's user avatar
  • 591
34 votes
5 answers
13k views

What are the use cases for Apache Spark vs Hadoop

With Hadoop 2.0 and YARN Hadoop is supposedly no longer tied only map-reduce solutions. With that advancement, what are the use cases for Apache Spark vs Hadoop considering both sit atop of HDFS? I've ...
idclark's user avatar
  • 521
15 votes
2 answers
4k views

What is the difference between Hadoop and noSQL

I heard about many tools / frameworks for helping people to process their data (big data environment). One is called Hadoop and the other is the noSQL concept. What is the difference in point of ...
рüффп's user avatar
12 votes
2 answers
7k views

Tradeoffs between Storm and Hadoop (MapReduce)

Can someone kindly tell me about the trade-offs involved when choosing between Storm and MapReduce in Hadoop Cluster for data processing? Of course, aside from the obvious one, that Hadoop (processing ...
mbbce's user avatar
  • 347
12 votes
3 answers
3k views

Does Amazon RedShift replace Hadoop for ~1XTB data?

There is plenty of hype surrounding Hadoop and its eco-system. However, in practice, where many data sets are in the terabyte range, is it not more reasonable to use Amazon RedShift for querying ...
trienism's user avatar
  • 253
11 votes
3 answers
1k views

What are R's memory constraints?

In reviewing “Applied Predictive Modeling" a reviewer states: One critique I have of statistical learning (SL) pedagogy is the absence of computation performance considerations in the evaluation ...
blunders's user avatar
  • 1,932
11 votes
3 answers
711 views

Can map-reduce algorithms written for MongoDB be ported to Hadoop later?

In our company, we have a MongoDB database containing a lot of unstructured data, on which we need to run map-reduce algorithms to generate reports and other analyses. We have two approaches to select ...
Amir Ali Akbari's user avatar
8 votes
4 answers
1k views

Data science and MapReduce programming model of Hadoop

What are the different classes of data science problems that can be solved using mapreduce programming model?
10land's user avatar
  • 369
8 votes
3 answers
4k views

Good books for Hadoop, Spark, and Spark Streaming [closed]

Can anyone suggest any good books to learn hadoop and map reduce basics? Also something for Spark, and Spark Streaming? Thanks
tsar2512's user avatar
  • 183
8 votes
1 answer
78 views

Cascaded Error in Apache Storm

Going through the presentation and material of Summingbird by Twitter, one of the reasons that is mentioned for using Storm and Hadoop clusters together in Summingbird is that processing through Storm ...
mbbce's user avatar
  • 347
7 votes
2 answers
3k views

Lambda Architecture - How to implement the Merge Layer / Query Layer

I am reading up about lambda architecture. It makes sense. we have queue based data ingestion. we have an in-memory store for data which is very new and we have HDFS for old data. So we have our ...
Knows Not Much's user avatar
7 votes
1 answer
1k views

Linear Regression in R Mapreduce(RHadoop)

I m new to RHadoop and also to RMR... I had an requirement to write a Mapreduce Job in R Mapreduce. I have Tried writing but While executing this it gives an Error. Tring to read the file from hdfs ...
user3782364's user avatar
6 votes
3 answers
2k views

Is there a benefit to using hadoop with only one node?

I just started learning about Hadoop. From what I understand its primary strength is its ability to distribute a task across many nodes. Is there any benefit to using Hadoop with only a single node, ...
Eric Anastas's user avatar
6 votes
1 answer
3k views

Improve k-means accuracy

Our weapons: I am experimenting with k-means and Hadoop, where I am chained to these options for various reasons (e.g. Help me win this war!). The battlefield: I have articles, which belong to c ...
gsamaras's user avatar
  • 291
6 votes
1 answer
3k views

How to make k-means distributed?

After setting up a 2-noded Hadoop cluster, understanding Hadoop and Python and based on this naive implementation, I ended up with this code: ...
gsamaras's user avatar
  • 291
6 votes
2 answers
1k views

Processing data stored in Redshift

We're currently using Redshift as our data warehouse, which we're very happy with. However, we now have a requirement to do machine learning against the data in our warehouse. Given the volume of data ...
deanj's user avatar
  • 63
5 votes
8 answers
660 views

Skills that school doesn't teach you

I'll be graduating soon with a master's degree in electrical engineering. I have a lot of experience with the theoretical side of machine learning through courses and research, but all of my ...
Sam's user avatar
  • 51
5 votes
4 answers
1k views

Can all statistical algorithms be parallelized using a Map Reduce framework

Is it correct to say that any statistical learning algorithm (linear/logistic regression, SVM, neural network, random forest) can be implemented inside a Map Reduce framework? Or are there ...
Victor's user avatar
  • 611
5 votes
3 answers
1k views

Storing Sensor Data for Analysis of the Office

I have currently been tasked with designing an application that tracks several different measurements around the office, eg. the temperature, light, presence of people, etc. Having never really worked ...
Utsav Tiwary's user avatar
5 votes
1 answer
2k views

Saving Large Spark ML Pipeline to HDFS

I'm having trouble saving a large (relative to spark.rpc.message.maxSize) Spark ML pipeline to HDFS. Specifically, when I try to save the model to HDFS, it gives me an error related to spark's maximum ...
Thomas Cleberg's user avatar
4 votes
2 answers
4k views

Can we access HDFS file system and YARN scheduler in Apache Spark?

We can access HDFS file system and YARN scheduler In the Apache-Hadoop. But Spark has a higher level of coding. Is it possible to access HDFS and YARN in Apache-Spark too? Thanks
Omid Ebrahimi's user avatar
4 votes
3 answers
808 views

Hadoop for grid computing

Background: I run a product that compares sets of data (data matching and data reconciliation). To get the result we need to compare each row in a data set with every N rows on the opposing data set ...
papacostas's user avatar
4 votes
1 answer
914 views

What is the main difference between Hadoop and Spark? [closed]

I recently read the following about Hadoop vs. Spark: Insist upon in-memory columnar data querying. This was the killer-feature that let Apache Spark run in seconds the queries that would take Hadoop ...
Ironclad's user avatar
4 votes
1 answer
404 views

Is there any point in learning Hadoop in 2018?

I'm a data analyst/scientist working mostly with the Python open source stack i.e. Pandas, scikit-learn, matplotlib, seaborn etc. I want to expand my toolbox and learn a distributed computing ...
RodrikTheReader's user avatar
4 votes
1 answer
2k views

Hive: How to calculate the Kendall coefficient of correlation of a pair of a numeric columns in the group?

In this wiki page there is a function corr() that calculates the Pearson coefficient of correlation, but my question is that: is there any function in Hive that ...
Marcin's user avatar
  • 235
4 votes
2 answers
100 views

Pig script code error?

While running the below pig script I am getting an error in line4: If it is GROUP then I am getting error. If I change from 'GROUP' TO 'group' in line4, then the script is running. What is the ...
Green's user avatar
  • 243
4 votes
1 answer
1k views

How to set up multi cluster spark without hadoop on Google Compute engine

I'm new to apache spark. Is it possible to configure multi cluster spark without hadoop? If so, can you please provide the steps. I would like to create clusters on Google Compute Engine (1-...
user4290511's user avatar
3 votes
2 answers
373 views

Is our data "Big Data" (Startup)

I worked at a startup/medium sized company and I am concerned that we may be over-engineering one of our products. In essence, we will be consuming real-time coordinates from vehicles and users and ...
Kyle's user avatar
  • 33
3 votes
2 answers
1k views

Cloudera QuickStart VM Error [closed]

I have installed cloudera CDH5 Quick start VM on VM player. When I login through HUE in the first page I am the following error “Potential misconfiguration detected. Fix and restart Hue.” How to ...
Green's user avatar
  • 243
3 votes
1 answer
557 views

BERT in production

I've created a BERT model. What are the ways to do the deployment of this model? Is it possible to use it with Spark, Hadoop or Docker?
illuminato's user avatar
3 votes
1 answer
263 views

Deploying models on bigdata platforms like Hadoop and Spark

Where exactly bigdata platforms fit in to a data science/ machine learning projects ? Say I have a large dataset for a binary classification problem - cats and dogs. Now I need to create a model for ...
Sreejithc321's user avatar
  • 1,920
3 votes
2 answers
12k views

Hadoop Resource Manager Won't Start [closed]

I am a relatively new user to Hadoop (using version 2.4.1). I installed hadoop on my first node without a hitch, but I can't seem to get the Resource Manager to start on my second node. I cleared up ...
BigDataDude's user avatar
3 votes
1 answer
887 views

HBase connector - Thrift or REST

There seem to be at least 2 ways to connect to HBase from external application, with language other then Java (i.e. Python): HBase Thrift API HBase Stargate (REST API) Does anyone know which one ...
Igor Bobriakov's user avatar
3 votes
1 answer
84 views

Data produced as an output to Dumbo API of Python not getting distributed to all the nodes of cluster

On the node from which I run Dumbo commands, all the files produced as output are produced on the same node. For example, suppose there is a node having name hvs on which I ran the script: dumbo ...
Harshvardhan Solanki's user avatar
3 votes
1 answer
3k views

java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream

Im traying to integrate Hadoop and R, I was install the pachages rJava and Rhipe in R, I do this steps to start Hadoop and R: -starting Hadoop services., -loading rJava and Rhipe packages by library ...
abdoBim's user avatar
  • 163
3 votes
0 answers
586 views

Communication between name node, data node and client in hadoop by analysing Packet Capturing [closed]

if I say that in hadoop cluster system... A client has 10GB data to send to name node. After analysing the "packet capturing" actually the name node doesn't divide the data for data node, ...
Radheyshyam Sain's user avatar
3 votes
0 answers
231 views

How to setup a home-laptop cluster to 'practice' elasticsearch, hadoop, mesos and spark [closed]

I'd like to have elasticsearch, hadoop, mesos and spark on my home laptop (it's an 8GB with an intel i7 CPU) for practicing the setup of such frameworks and eventually to test their functions. I'm ...
user1403546's user avatar
3 votes
1 answer
114 views

Pig latin code error [duplicate]

While running the below pig script I am getting error in line4: If it is GROUP then I am getting error. If I change from GROUP ...
Green's user avatar
  • 243
3 votes
2 answers
26k views

How to read contents of a CSV file inside zip file using spark (python) [closed]

I have multiple zip files containing two types of files(A.csv & B.csv) /data/jan.zip --> contains A.csv & B.csv /data/feb.zip --> contains A.csv & B.csv I want to read the contents of ...
Munesh's user avatar
  • 131
2 votes
2 answers
6k views

spark item similarity recommendation

I would like to build an recommendation engine using spark's Mlib itemsimilarity as mentioned here LINK But it seems spark do not have this algorithm any more and ...
Dimag Kharab's user avatar
2 votes
2 answers
667 views

Why has Hadoop failed to become popular?

Apache Hadoop was once considered one of the tools you should have as a Data Scientist. Around 2012 to 2014, it gained the popularity steeply among Data Scientists, and seemed to be considered one of ...
Blaszard's user avatar
  • 911
2 votes
3 answers
234 views

Can Hadoop be beneficial when data is in database tables and not in a file system

I work for a bank. Most of our data is in the form of database tables. Would we benefit by implementing Hadoop? I am of the impression that Hadoop is more for a Distributed File System (unstructured ...
Victor's user avatar
  • 611
2 votes
3 answers
364 views

Differences in scoring from PMML model on different platforms

I've built a toy Random Forest model in R (using the German Credit dataset from the caret ...
neuron's user avatar
  • 664
2 votes
2 answers
10k views

PySpark: java.io.EOFException

System: 1 name node, 4 cores, 16 GB RAM 1 master node, 4 cores, 16 GB RAM 6 data nodes, 4 cores, 16 GB RAM each 6 worker nodes, 4 cores, 16 GB RAM each around 5 Terabytes of storage space The data ...
dustin's user avatar
  • 131
2 votes
1 answer
70 views

does storing file in hdfs parallelize it for Spark?

For Spark's RDD operations, data must be in shape of RDD or be parallelized using: ParallelizedData = sc.parallelize(data) My question is that if I store data in ...
Ali Majed HA's user avatar
2 votes
1 answer
1k views

Ingestion of periodic REST API Calls into Hadoop [closed]

We are planning to use REST API calls to ingest data from an endpoint and store the data to HDFS. The REST calls are done in a periodic fashion (daily or maybe hourly). I've already done Twitter ...
geek-tech's user avatar
2 votes
2 answers
144 views

How many people can use a single Hadoop cluster at one time?

I would like to know how many people can use a single Hadoop cluster at one time? I am asking because I need to figure out whether or not a single 5 or 10 node cluster would be sufficient to host a ...
eglinker's user avatar
2 votes
1 answer
862 views

Difference Between Hadoop Mapreduce(Java) and RHadoop mapreduce

I understand Hadoop MapReduce and its features but I am confused about R MapReduce. One difference I have read is that R utilizes maximum RAM. So do perform parallel processing integrated R with ...
user3782364's user avatar
2 votes
1 answer
6k views

Are parquet files compressed? [closed]

Parquet File Format Hadoop. Parquet, an open-source file format for Hadoop. Parquet stores nested data structures in a flat columnar format. Compared to a traditional approach where data is stored in ...
Pluviophile's user avatar
  • 3,958
2 votes
2 answers
3k views

Accumulators in Spark (PySpark) without global variables?

BACKGROUND Consider the following textbook example which uses accumulators to add vectors. ...
Tfovid's user avatar
  • 195