Questions tagged [apache-hadoop]
Hadoop is an Apache open-source project that provides software for reliable and scalable distributed computing. The project itself includes a variety of other complementary additions.
118
questions
40
votes
10
answers
21k
views
Do I need to learn Hadoop to be a Data Scientist?
An aspiring data scientist here. I don't know anything about Hadoop, but as I have been reading about Data Science and Big Data, I see a lot of talk about Hadoop. Is it absolutely necessary to learn ...
- 591
34
votes
5
answers
13k
views
What are the use cases for Apache Spark vs Hadoop
With Hadoop 2.0 and YARN Hadoop is supposedly no longer tied only map-reduce solutions. With that advancement, what are the use cases for Apache Spark vs Hadoop considering both sit atop of HDFS? I've ...
- 521
15
votes
2
answers
4k
views
What is the difference between Hadoop and noSQL
I heard about many tools / frameworks for helping people to process their data (big data environment).
One is called Hadoop and the other is the noSQL concept. What is the difference in point of ...
- 295
12
votes
2
answers
7k
views
Tradeoffs between Storm and Hadoop (MapReduce)
Can someone kindly tell me about the trade-offs involved when choosing between Storm and MapReduce in Hadoop Cluster for data processing? Of course, aside from the obvious one, that Hadoop (processing ...
- 347
12
votes
3
answers
3k
views
Does Amazon RedShift replace Hadoop for ~1XTB data?
There is plenty of hype surrounding Hadoop and its eco-system. However, in practice, where many data sets are in the terabyte range, is it not more reasonable to use Amazon RedShift for querying ...
- 253
11
votes
3
answers
1k
views
What are R's memory constraints?
In reviewing “Applied Predictive Modeling" a reviewer states:
One critique I have of statistical learning (SL) pedagogy is the
absence of computation performance considerations in the evaluation ...
- 1,932
11
votes
3
answers
704
views
Can map-reduce algorithms written for MongoDB be ported to Hadoop later?
In our company, we have a MongoDB database containing a lot of unstructured data, on which we need to run map-reduce algorithms to generate reports and other analyses. We have two approaches to select ...
- 1,393
8
votes
4
answers
1k
views
Data science and MapReduce programming model of Hadoop
What are the different classes of data science problems that can be solved using mapreduce programming model?
- 369
8
votes
3
answers
4k
views
Good books for Hadoop, Spark, and Spark Streaming [closed]
Can anyone suggest any good books to learn hadoop and map reduce basics?
Also something for Spark, and Spark Streaming?
Thanks
- 183
8
votes
1
answer
75
views
Cascaded Error in Apache Storm
Going through the presentation and material of Summingbird by Twitter, one of the reasons that is mentioned for using Storm and Hadoop clusters together in Summingbird is that processing through Storm ...
- 347
7
votes
2
answers
3k
views
Lambda Architecture - How to implement the Merge Layer / Query Layer
I am reading up about lambda architecture.
It makes sense. we have queue based data ingestion. we have an in-memory store for data which is very new and we have HDFS for old data.
So we have our ...
- 171
7
votes
1
answer
1k
views
Linear Regression in R Mapreduce(RHadoop)
I m new to RHadoop and also to RMR...
I had an requirement to write a Mapreduce Job in R Mapreduce. I have Tried writing but While executing this it gives an Error.
Tring to read the file from hdfs
...
- 101
6
votes
3
answers
2k
views
Is there a benefit to using hadoop with only one node?
I just started learning about Hadoop. From what I understand its primary strength is its ability to distribute a task across many nodes. Is there any benefit to using Hadoop with only a single node, ...
- 161
6
votes
1
answer
3k
views
Improve k-means accuracy
Our weapons:
I am experimenting with k-means and Hadoop, where I am chained to these options for various reasons (e.g. Help me win this war!).
The battlefield:
I have articles, which belong to c ...
- 291
6
votes
1
answer
3k
views
How to make k-means distributed?
After setting up a 2-noded Hadoop cluster, understanding Hadoop and Python and based on this naive implementation, I ended up with this code:
...
- 291
6
votes
2
answers
1k
views
Processing data stored in Redshift
We're currently using Redshift as our data warehouse, which we're very happy with. However, we now have a requirement to do machine learning against the data in our warehouse. Given the volume of data ...
- 63
5
votes
8
answers
653
views
Skills that school doesn't teach you
I'll be graduating soon with a master's degree in electrical engineering. I have a lot of experience with the theoretical side of machine learning through courses and research, but all of my ...
- 51
5
votes
4
answers
1k
views
Can all statistical algorithms be parallelized using a Map Reduce framework
Is it correct to say that any statistical learning algorithm (linear/logistic regression, SVM, neural network, random forest) can be implemented inside a Map Reduce framework? Or are there ...
- 611
5
votes
3
answers
1k
views
Storing Sensor Data for Analysis of the Office
I have currently been tasked with designing an application that tracks several different measurements around the office, eg. the temperature, light, presence of people, etc. Having never really worked ...
- 53
5
votes
1
answer
2k
views
Saving Large Spark ML Pipeline to HDFS
I'm having trouble saving a large (relative to spark.rpc.message.maxSize) Spark ML pipeline to HDFS. Specifically, when I try to save the model to HDFS, it gives me an error related to spark's maximum ...
- 1,515
4
votes
2
answers
4k
views
Can we access HDFS file system and YARN scheduler in Apache Spark?
We can access HDFS file system and YARN scheduler In the Apache-Hadoop. But Spark has a higher level of coding. Is it possible to access HDFS and YARN in Apache-Spark too?
Thanks
- 249
4
votes
3
answers
786
views
Hadoop for grid computing
Background:
I run a product that compares sets of data (data matching and data reconciliation).
To get the result we need to compare each row in a data set with every N rows on the opposing data set
...
- 141
4
votes
1
answer
402
views
Is there any point in learning Hadoop in 2018?
I'm a data analyst/scientist working mostly with the Python open source stack i.e. Pandas, scikit-learn, matplotlib, seaborn etc. I want to expand my toolbox and learn a distributed computing ...
- 141
4
votes
1
answer
2k
views
Hive: How to calculate the Kendall coefficient of correlation of a pair of a numeric columns in the group?
In this wiki page there is a function corr() that calculates the Pearson coefficient of correlation, but my question is that: is there any function in Hive that ...
- 235
4
votes
2
answers
97
views
Pig script code error?
While running the below pig script I am getting an error in line4:
If it is GROUP then I am getting error.
If I change from 'GROUP' TO 'group' in line4, then the script is running.
What is the ...
- 243
4
votes
1
answer
1k
views
How to set up multi cluster spark without hadoop on Google Compute engine
I'm new to apache spark.
Is it possible to configure multi cluster spark without hadoop?
If so, can you please provide the steps.
I would like to create clusters on Google Compute Engine (1-...
- 101
3
votes
2
answers
366
views
Is our data "Big Data" (Startup)
I worked at a startup/medium sized company and I am concerned that we may be over-engineering one of our products.
In essence, we will be consuming real-time coordinates from vehicles and users and ...
- 33
3
votes
2
answers
1k
views
Cloudera QuickStart VM Error [closed]
I have installed cloudera CDH5 Quick start VM on VM player. When I login through HUE in the first page I am the following error
“Potential misconfiguration detected. Fix and restart Hue.”
How to ...
- 243
3
votes
1
answer
588
views
What is the main difference between Hadoop and Spark? [closed]
I recently read the following about Hadoop vs. Spark:
Insist upon in-memory columnar data querying. This was the killer-feature that let Apache Spark run in seconds the queries that would take Hadoop ...
- 31
3
votes
1
answer
489
views
BERT in production
I've created a BERT model. What are the ways to do the deployment of this model? Is it possible to use it with Spark, Hadoop or Docker?
- 308
3
votes
1
answer
262
views
Deploying models on bigdata platforms like Hadoop and Spark
Where exactly bigdata platforms fit in to a data science/ machine learning projects ?
Say I have a large dataset for a binary classification problem - cats and dogs.
Now I need to create a model for ...
- 1,900
3
votes
2
answers
12k
views
Hadoop Resource Manager Won't Start [closed]
I am a relatively new user to Hadoop (using version 2.4.1). I installed hadoop on my first node without a hitch, but I can't seem to get the Resource Manager to start on my second node.
I cleared up ...
- 91
3
votes
1
answer
875
views
HBase connector - Thrift or REST
There seem to be at least 2 ways to connect to HBase from external application, with language other then Java (i.e. Python):
HBase Thrift API
HBase Stargate (REST API)
Does anyone know which one ...
- 1,071
3
votes
1
answer
81
views
Data produced as an output to Dumbo API of Python not getting distributed to all the nodes of cluster
On the node from which I run Dumbo commands, all the files produced as output are produced on the same node. For example, suppose there is a node having name hvs on which I ran the script:
dumbo ...
3
votes
1
answer
3k
views
java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
Im traying to integrate Hadoop and R, I was install the pachages rJava and Rhipe in R, I do this steps to start Hadoop and R:
-starting Hadoop services.,
-loading rJava and Rhipe packages by library ...
- 163
3
votes
0
answers
580
views
Communication between name node, data node and client in hadoop by analysing Packet Capturing [closed]
if I say that in hadoop cluster system... A client has 10GB data to send to name node. After analysing the "packet capturing" actually the name node doesn't divide the data for data node, ...
3
votes
0
answers
230
views
How to setup a home-laptop cluster to 'practice' elasticsearch, hadoop, mesos and spark [closed]
I'd like to have elasticsearch, hadoop, mesos and spark on my home laptop (it's an 8GB with an intel i7 CPU) for practicing the setup of such frameworks and eventually to test their functions.
I'm ...
- 153
3
votes
1
answer
112
views
Pig latin code error [duplicate]
While running the below pig script I am getting error in line4: If it is GROUP then I am getting error. If I change from GROUP ...
- 243
3
votes
2
answers
25k
views
How to read contents of a CSV file inside zip file using spark (python) [closed]
I have multiple zip files containing two types of files(A.csv & B.csv)
/data/jan.zip --> contains A.csv & B.csv
/data/feb.zip --> contains A.csv & B.csv
I want to read the contents of ...
- 131
2
votes
2
answers
6k
views
spark item similarity recommendation
I would like to build an recommendation engine using spark's Mlib itemsimilarity as mentioned here LINK
But it seems spark do not have this algorithm any more and ...
- 141
2
votes
2
answers
662
views
Why has Hadoop failed to become popular?
Apache Hadoop was once considered one of the tools you should have as a Data Scientist. Around 2012 to 2014, it gained the popularity steeply among Data Scientists, and seemed to be considered one of ...
- 901
2
votes
3
answers
208
views
Can Hadoop be beneficial when data is in database tables and not in a file system
I work for a bank. Most of our data is in the form of database tables. Would we benefit by implementing Hadoop? I am of the impression that Hadoop is more for a Distributed File System (unstructured ...
- 611
2
votes
3
answers
359
views
Differences in scoring from PMML model on different platforms
I've built a toy Random Forest model in R (using the German Credit dataset from the caret ...
- 664
2
votes
2
answers
8k
views
PySpark: java.io.EOFException
System:
1 name node, 4 cores, 16 GB RAM
1 master node, 4 cores, 16 GB RAM
6 data nodes, 4 cores, 16 GB RAM each
6 worker nodes, 4 cores, 16 GB RAM each
around 5 Terabytes of storage space
The data ...
- 131
2
votes
1
answer
69
views
does storing file in hdfs parallelize it for Spark?
For Spark's RDD operations, data must be in shape of RDD or be parallelized using:
ParallelizedData = sc.parallelize(data)
My question is that if I store data in ...
- 143
2
votes
1
answer
1k
views
Ingestion of periodic REST API Calls into Hadoop [closed]
We are planning to use REST API calls to ingest data from an endpoint and store the data to HDFS. The REST calls are done in a periodic fashion (daily or maybe hourly).
I've already done Twitter ...
- 29
2
votes
2
answers
139
views
How many people can use a single Hadoop cluster at one time?
I would like to know how many people can use a single Hadoop cluster at one time? I am asking because I need to figure out whether or not a single 5 or 10 node cluster would be sufficient to host a ...
- 21
2
votes
1
answer
860
views
Difference Between Hadoop Mapreduce(Java) and RHadoop mapreduce
I understand Hadoop MapReduce and its features but I am confused about R MapReduce.
One difference I have read is that R utilizes maximum RAM. So do perform parallel processing integrated R with ...
- 101
2
votes
1
answer
5k
views
Are parquet files compressed? [closed]
Parquet File Format Hadoop. Parquet, an open-source file format for Hadoop. Parquet stores nested data structures in a flat columnar format. Compared to a traditional approach where data is stored in ...
- 3,608
2
votes
2
answers
3k
views
Accumulators in Spark (PySpark) without global variables?
BACKGROUND
Consider the following textbook example which uses accumulators to add vectors.
...
- 203