Questions tagged [apache-hadoop]

Hadoop is an Apache open-source project that provides software for reliable and scalable distributed computing. The project itself includes a variety of other complementary additions.

Filter by
Sorted by
Tagged with
0 votes
1 answer
70 views

Which one of these tasks will benefit the most from SPARK?

My company processes Data (I am only an intern). We primarily use Hadoop. We're starting to deploy spark in production. Currently we have have two jobs, we will choose just one to begin with spark. ...
Melchia's user avatar
  • 111
0 votes
1 answer
151 views

Install Spark and Hadoop in the same machine [closed]

I'm trying to install Spark Apache and Hadoop in the same machine. Spark will be used to process data, and HDFS from Hadoop will be used to store data. I first started by installing spark and it ...
Mehdi Ben Hamida's user avatar
0 votes
1 answer
2k views

Machine Learning model to find items that are frequently bought together using Hadoop Spark

I need to do my master thesis under Big Data Analytics - Machine Learning subject. I was challenged to create a Market Basket Analysis project on retail industry. My dataset contains transactions of ...
João_testeSW's user avatar
0 votes
1 answer
292 views

Spark algorithm to make a link analysis

I'm doing my master thesis on Big Data Analytics. I'm trying to develop a algorithm to identify associations between some products in a supermakert. Imagine that I've this dataset: ...
João_testeSW's user avatar
0 votes
3 answers
278 views

Predictive Analytics on distributed systems vs standalone system

I have somewhat philosophical question regarding performing predictive analytics on distributed systems (such as hadoop). I am no expert on this subject so maybe other folks who have more advance ...
buy_sell1's user avatar
0 votes
1 answer
57 views

Accessing directory of small files as one file

I have a job that results in a directory of part- files. I'd like to read it as if it were one file. Specifically, I'd like to read it that way over a web interface. How can I do either one of these ...
Marcin's user avatar
  • 101
0 votes
1 answer
312 views

When it is time to use Hadoop? [closed]

Hadoop is a buzzword now. A lot of start-ups use it (or just say, that they use it), a lot of widely known companies use it. But when and what is the border? When person can say: "Better to solve it ...
Aleksandro M Granda's user avatar
0 votes
1 answer
93 views

Questions on "Active Archive"

In the big data world there is lot of talk about implementing an "active archive". I see cloudera talk about it a lot. The value prop is that you move the low value (and less used) data from EDW to ...
Knows Not Much's user avatar
0 votes
1 answer
60 views

Hadoop, Spark and Cloud

It seems Hadoop, Spark, and different versions of Clouds offer facilities to store and analyze big data. There are some articles comparing Hadoop and Spark (for example, this article). There are also ...
Tara's user avatar
  • 1
0 votes
1 answer
35 views

What is the difference between Data Modeling and Data Processing?

When discussing big data, it is sometimes mentioned that data modeling can be done by using a tool like map reduce, while data processing may be performed by apache spark. What is the difference ...
Karl 17302's user avatar
0 votes
1 answer
263 views

Storage of N-dimensional matrices (tensors) as part of machine learning pipelines

I'm an infra person working on a storage product. I've been googling quite a bit to find an answer to the following question but unable to do so. Hence, I am attemping to ask the question here. I am ...
user855's user avatar
  • 101
0 votes
1 answer
31 views

Can Single Node Hadoop Cluster be installed on a system with 1gb RAM

I am trying to learn hadoop, would like to know if for basic single node installation 1gb RAM system would be enough or we need more RAM. It would be helpful if someone can share what other minimum ...
Gaurav Parek's user avatar
0 votes
1 answer
373 views

Hive / Impala best practice code structuring

Coming from a DWH-background I am used to putting subqueries almost everywhere in my queries. On a Hadoop project (with Hive version 1.1.0 on Cloudera), I noticed we can forego subqueries in some ...
Gerardsson's user avatar
0 votes
1 answer
23 views

What are common problems around HADOOP storage?

I've been asked to lead a program to understand why our Hadoop storage is constantly near capacity. What questions should I ask? Data age, Data size? Housekeeping schedule? How do we identify the ...
vwdewaal's user avatar
  • 101
0 votes
1 answer
150 views

Yarn timeline recovery not enabled error upgrading via ambari

Using the automated upgrade, when I try to upgrade I get: “YARN Timeline state preserving restart should be enabled Reason: YARN should have state preserving restart enabled for the Timeline server. ...
Chris B-C's user avatar
  • 101
0 votes
1 answer
60 views

SAP HANA or Hadoop? [closed]

this is a question regarding a career choice. I am a fresher and I recently joined an MNC in Data Engineering team. There I was offered training in either Hadoop or SAP HANA. I am in doubt as to which ...
AswinRajaram's user avatar
-1 votes
1 answer
95 views

Connecting Twitter API to a Big Data Environment?

I am trying to get live twitter data and store it in a Hadoop environment. Currently, I am learning how to push the data into a Cloudera environment and process it. My questions are: Is Cloudera ...
Melvin Roy's user avatar
-2 votes
1 answer
4k views

getting error:-Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/io/Writable [closed]

I am trying to connect to hive from java but getting error. I searched in google but not got any helpfull solution. I have added all jars also. The code is:- ...
Jaishree Rout's user avatar

1 2
3