Questions tagged [apache-hadoop]
Hadoop is an Apache open-source project that provides software for reliable and scalable distributed computing. The project itself includes a variety of other complementary additions.
119
questions
2
votes
1
answer
1k
views
Ingestion of periodic REST API Calls into Hadoop [closed]
We are planning to use REST API calls to ingest data from an endpoint and store the data to HDFS. The REST calls are done in a periodic fashion (daily or maybe hourly).
I've already done Twitter ...
- 29
2
votes
1
answer
242
views
K-means clustering on big data stored on multiple nodes on HDFS
I have a very basic question about how the k-means clustering algorithm works when the data is stored in HDFS.
Does the training take place on the subset of data on each node for a fixed value of K( ...
- 977
2
votes
2
answers
139
views
How many people can use a single Hadoop cluster at one time?
I would like to know how many people can use a single Hadoop cluster at one time? I am asking because I need to figure out whether or not a single 5 or 10 node cluster would be sufficient to host a ...
- 21
2
votes
1
answer
406
views
Mahout Spark shell not working
I installed Hadoop, Mahout and Spark. I am able to see the Hadoop and Spark MasterWebUI. Moreover, I can also run the following command,
...
- 141
2
votes
2
answers
6k
views
spark item similarity recommendation
I would like to build an recommendation engine using spark's Mlib itemsimilarity as mentioned here LINK
But it seems spark do not have this algorithm any more and ...
- 141
1
vote
1
answer
185
views
How to Scaling Out Artifical Neural Networks?
I have started to study ANNs with Tensorflow and Keras. Now I want to find a solution to use ANNs over Hadoop. I have learnt that Spark 2.0 does have a Multilayer Perceptron Classifier, but as far as ...
- 8,457
-1
votes
1
answer
93
views
Connecting Twitter API to a Big Data Environment?
I am trying to get live twitter data and store it in a Hadoop environment. Currently, I am learning how to push the data into a Cloudera environment and process it.
My questions are:
Is Cloudera ...
0
votes
1
answer
2k
views
Machine Learning model to find items that are frequently bought together using Hadoop Spark
I need to do my master thesis under Big Data Analytics - Machine Learning subject. I was challenged to create a Market Basket Analysis project on retail industry.
My dataset contains transactions of ...
- 179
0
votes
1
answer
291
views
Spark algorithm to make a link analysis
I'm doing my master thesis on Big Data Analytics. I'm trying to develop a algorithm to identify associations between some products in a supermakert. Imagine that I've this dataset:
...
- 179
5
votes
8
answers
653
views
Skills that school doesn't teach you
I'll be graduating soon with a master's degree in electrical engineering. I have a lot of experience with the theoretical side of machine learning through courses and research, but all of my ...
- 51
0
votes
3
answers
277
views
Predictive Analytics on distributed systems vs standalone system
I have somewhat philosophical question regarding performing predictive analytics on distributed systems (such as hadoop). I am no expert on this subject so maybe other folks who have more advance ...
1
vote
0
answers
175
views
How to multiply a "fat and short" matrix with a "tall and thin" matrix using MapReduce?
Assume that $A_{m \times n}$ and $B_{n \times k}$ are to be multiplied to get $C_{m \times k}$. Now if $n$ is too large for a single row $A_{j}$ of $A$ to fit in RAM (and similarly for columns of B) ...
- 1,297
1
vote
1
answer
114
views
Suggestions on what patterns/analysis to derive from Airlines Big Data [closed]
I recently started learning Hadoop,
I found this data set http://stat-computing.org/dataexpo/2009/the-data.html - (2009 data),
I want some suggestions as what type of patterns or analysis can I do in ...
- 191
1
vote
1
answer
65
views
Is there a text on Apache Spark that attempts to be as comprehensive as White's Hadoop: The Definitive Guide'?
Tom White's 'Hadoop: the Definitive Guide' has become a popular guide to the entire Hadoop ecosystem and earned a reputation as providing both a broad survey, as well as covering individual aspects of ...
- 899
2
votes
2
answers
595
views
3
votes
2
answers
25k
views
How to read contents of a CSV file inside zip file using spark (python) [closed]
I have multiple zip files containing two types of files(A.csv & B.csv)
/data/jan.zip --> contains A.csv & B.csv
/data/feb.zip --> contains A.csv & B.csv
I want to read the contents of ...
- 131
1
vote
0
answers
118
views
freebcp getting stalled for huge data
I am working on a project which processes big data (size ~3TB) every day. The first stage of our data pipeline copies the data from our MS-SQL Server to a host machine(linux server) using a tool ...
- 111
6
votes
1
answer
3k
views
How to make k-means distributed?
After setting up a 2-noded Hadoop cluster, understanding Hadoop and Python and based on this naive implementation, I ended up with this code:
...
- 291
6
votes
1
answer
3k
views
Improve k-means accuracy
Our weapons:
I am experimenting with k-means and Hadoop, where I am chained to these options for various reasons (e.g. Help me win this war!).
The battlefield:
I have articles, which belong to c ...
- 291
1
vote
1
answer
1k
views
A Simple Explanation of ZooKeeper in Hadoop
Can Anybody Explain me the role of ZooKeeper and ZooKeeper Quorum in a BigData structure?
I'd appreciate if you explain me with a simple practical example instead of abstract theoretical definitions.
...
- 163
1
vote
0
answers
259
views
Pig is not able to read the complete data
I am trying to load a huge dataset of around 3.4 TB with approximately 1.4 million files in Pig on Amazon EMR.
The operations on the data are simple (JOIN and STORE), but the data is not getting ...
- 373
1
vote
1
answer
6k
views
how to disable query from beeline results
I am occuring a strange hive-client beeline behavior.
In the outputed file with query results there is also a queary at the ...
- 235
0
votes
1
answer
148
views
Yarn timeline recovery not enabled error upgrading via ambari
Using the automated upgrade, when I try to upgrade I get:
“YARN Timeline state preserving restart should be enabled
Reason: YARN should have state preserving restart enabled for the Timeline server. ...
- 101
6
votes
3
answers
2k
views
Is there a benefit to using hadoop with only one node?
I just started learning about Hadoop. From what I understand its primary strength is its ability to distribute a task across many nodes. Is there any benefit to using Hadoop with only a single node, ...
- 161
2
votes
3
answers
204
views
Can Hadoop be beneficial when data is in database tables and not in a file system
I work for a bank. Most of our data is in the form of database tables. Would we benefit by implementing Hadoop? I am of the impression that Hadoop is more for a Distributed File System (unstructured ...
- 611
5
votes
4
answers
1k
views
Can all statistical algorithms be parallelized using a Map Reduce framework
Is it correct to say that any statistical learning algorithm (linear/logistic regression, SVM, neural network, random forest) can be implemented inside a Map Reduce framework? Or are there ...
- 611
3
votes
2
answers
366
views
Is our data "Big Data" (Startup)
I worked at a startup/medium sized company and I am concerned that we may be over-engineering one of our products.
In essence, we will be consuming real-time coordinates from vehicles and users and ...
- 33
0
votes
1
answer
57
views
Accessing directory of small files as one file
I have a job that results in a directory of part- files. I'd like to read it as if it were one file. Specifically, I'd like to read it that way over a web interface.
How can I do either one of these ...
- 101
1
vote
0
answers
569
views
How to use REST API to execute Map-Reduce Task?
Let's say I have one Text File (Size 1 GB). I want to search particular word from file and if it is found the Line number should be returned.
I can Execute my java program using command line in linux....
- 131
5
votes
3
answers
1k
views
Storing Sensor Data for Analysis of the Office
I have currently been tasked with designing an application that tracks several different measurements around the office, eg. the temperature, light, presence of people, etc. Having never really worked ...
- 53
3
votes
1
answer
81
views
Data produced as an output to Dumbo API of Python not getting distributed to all the nodes of cluster
On the node from which I run Dumbo commands, all the files produced as output are produced on the same node. For example, suppose there is a node having name hvs on which I ran the script:
dumbo ...
1
vote
1
answer
144
views
Can we use HDFS and big data Analytics for processing huge log files being processed through some application on some central server?
Detailed Question Explanation:
Suppose say our application X is processing huge logs (size varying from MBs to GBs) and giving insight results in these logs(NOT A Social Data logs or Security Logs)
...
- 131
-2
votes
1
answer
4k
views
getting error:-Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable [closed]
I am trying to connect to hive from java but getting error. I searched in google but not got any helpfull solution. I have added all jars also.
The code is:-
...
- 165
2
votes
0
answers
1k
views
How append works in hdfs? Where the newly created instance of file is placed?
I'm a Newbie to Hadoop!
By web search and after going through hadoop guides, it is clear that hdfs doesn't allow us to edit the file but to append some data to existing file, it will be creating a ...
- 21
0
votes
1
answer
310
views
When it is time to use Hadoop? [closed]
Hadoop is a buzzword now. A lot of start-ups use it (or just say, that they use it), a lot of widely known companies use it. But when and what is the border? When person can say: "Better to solve it ...
0
votes
1
answer
5k
views
Extract company names/job titles from free text
I have a complete Hadoop platform with HDFS, MR, Hive, PIG, Hbase, etc., Python, R, Java. All data sets have a large size.
The data set A, describing the jobs of people working in a company, is ...
- 151
2
votes
1
answer
2k
views
Error when using MAX in Apache Pig (Hadoop)
I am trying to calculate maximum values for different groups in a relation in Pig. The relation has three columns patientid, featureid and featurevalue (all int).
I group the relation based on ...
- 131
4
votes
2
answers
4k
views
Can we access HDFS file system and YARN scheduler in Apache Spark?
We can access HDFS file system and YARN scheduler In the Apache-Hadoop. But Spark has a higher level of coding. Is it possible to access HDFS and YARN in Apache-Spark too?
Thanks
- 249
0
votes
1
answer
93
views
Questions on "Active Archive"
In the big data world there is lot of talk about implementing an "active archive". I see cloudera talk about it a lot.
The value prop is that you move the low value (and less used) data from EDW to ...
- 171
7
votes
2
answers
3k
views
Lambda Architecture - How to implement the Merge Layer / Query Layer
I am reading up about lambda architecture.
It makes sense. we have queue based data ingestion. we have an in-memory store for data which is very new and we have HDFS for old data.
So we have our ...
- 171
1
vote
0
answers
216
views
Hadoop/Pig Aggregate Data
I am working on a project with two data sets. A time vs. speed data set (let's call it traffic), and a time vs. weather data set (called weather).
I am looking to find a correlation between these ...
- 91
3
votes
1
answer
3k
views
java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
Im traying to integrate Hadoop and R, I was install the pachages rJava and Rhipe in R, I do this steps to start Hadoop and R:
-starting Hadoop services.,
-loading rJava and Rhipe packages by library ...
- 163
4
votes
1
answer
1k
views
How to set up multi cluster spark without hadoop on Google Compute engine
I'm new to apache spark.
Is it possible to configure multi cluster spark without hadoop?
If so, can you please provide the steps.
I would like to create clusters on Google Compute Engine (1-...
- 101
0
votes
1
answer
1k
views
Can hadoop with Spark be configured with 1GB RAM
I'm trying to set up a cluster (1 namenode, 1 datanode) on AWS.
I'm using free one year trial period of AWS, but the challenge is, instance is created with 1GB of RAM.
As I'm a student, I cannot ...
- 101
8
votes
3
answers
4k
views
Good books for Hadoop, Spark, and Spark Streaming [closed]
Can anyone suggest any good books to learn hadoop and map reduce basics?
Also something for Spark, and Spark Streaming?
Thanks
- 183
4
votes
1
answer
2k
views
Hive: How to calculate the Kendall coefficient of correlation of a pair of a numeric columns in the group?
In this wiki page there is a function corr() that calculates the Pearson coefficient of correlation, but my question is that: is there any function in Hive that ...
- 235
6
votes
2
answers
1k
views
Processing data stored in Redshift
We're currently using Redshift as our data warehouse, which we're very happy with. However, we now have a requirement to do machine learning against the data in our warehouse. Given the volume of data ...
- 63
1
vote
1
answer
8k
views
Masters thesis topics in big data
I am looking for a thesis to complete my master M2, I will work on a topic in the big data's field (creation big data applications), using hadoop/mapReduce and Ecosystem ( visualisation, analysis ...),...
- 163
2
votes
3
answers
359
views
Differences in scoring from PMML model on different platforms
I've built a toy Random Forest model in R (using the German Credit dataset from the caret ...
- 664
4
votes
3
answers
786
views
Hadoop for grid computing
Background:
I run a product that compares sets of data (data matching and data reconciliation).
To get the result we need to compare each row in a data set with every N rows on the opposing data set
...
- 141