Skip to main content
7 votes

spark item similarity recommendation

For your recommendation engine, if you've chosen to go by item similarity approach, then you can use Spark's RowMatrix datatype to achieve this task. Item similarity approach is just about creating ...
Santoshi M's user avatar
3 votes

Why has Hadoop failed to become popular?

You are comparing apples with oranges. Hadoop is one of the backend of big data platform and Python/R are programming languages which are used built prediction models and data pipeline. Hadoop still ...
Xformer's user avatar
  • 141
3 votes

Why has Hadoop failed to become popular?

I wouldn't say Hadoop failed to become popular rather I would say, it is still the base of any production big data system. Python or R are handy at the beginning when you just need to try out things ...
Tanmay Deshpande's user avatar
3 votes

Skills that school doesn't teach you

Based on my own experience and in reading what others have written, SQL is one of those skills employers look for, perhaps even assume that you have along with some of the basic skills of ...
jab's user avatar
  • 61
3 votes

Is there any point in learning Hadoop in 2018?

At this point of time, if I had to start a project from scratch and I had to choose between Hadoop and Spark, I would certainly choose Spark over Hadoop. There are several reasons for this: Spark is ...
Pablo Suau's user avatar
  • 1,787
3 votes

What is the main difference between Hadoop and Spark?

Hadoop is a framework for the distributed storage and processing of big data on the Hadoop File System (HDFS) where data is stored in a cluster of "nodes" and can be set up to be fault ...
Robert Long's user avatar
2 votes
Accepted

Suggestions on what patterns/analysis to derive from Airlines Big Data

There really is no wrong answer here, but I recommend predicting flight cancellations (#22) and/or delays (25-29), since this is how I often see this data set being used. It could also have practical ...
Ryan Zotti's user avatar
  • 4,149
2 votes
Accepted

Is there a text on Apache Spark that attempts to be as comprehensive as White's Hadoop: The Definitive Guide'?

Learning Spark: Lightning Fast Big Data Analytics is a fairly comprehensive book covering the core concepts as well as the higher level components involved in the Spark stack. This is the book ...
Vinay's user avatar
  • 154
2 votes

Skills that school doesn't teach you

If you find learning SQL and Hadoop unbearably boring, you should not be looking for a data scientist job. Anyhow, feel free to skip Hadoop. There are lots of deployments of Hadoop, but they are being ...
hajons's user avatar
  • 21
2 votes

Skills that school doesn't teach you

A large component of data science work in industry is data wrangling. It is quite important to have some basic understanding of data storage systems as you will often have to extract the data you need ...
0_0's user avatar
  • 965
2 votes

Skills that school doesn't teach you

I believe technology is cheap and science is expensive. You can learn R, Python, SQL and Hadoop pretty fast (considering that you know programming) but learning statistics, machine learning and the ...
Hamideh's user avatar
  • 940
2 votes

spark item similarity recommendation

There is spark-itemsimilarity command line tool that is based on Spark and Mahout. (It is not a library you import inside a Spark application.) Here is an ...
oW_'s user avatar
  • 6,367
2 votes

Deploying models on bigdata platforms like Hadoop and Spark

1 Since dataset is huge I can make use any disributed platform for faster computaion and model creation right? Yes, that is what distributed platforms are for. 2 Once the model is ready, then ...
Santoshi M's user avatar
2 votes

Ingestion of periodic REST API Calls into Hadoop

You can use Kafka to ingest data into HDFS or any other cloud storage like S3 or Google storage. And you can use Gobblin to schedule your kafka consumer to write into HDFS. ...
Abhis's user avatar
  • 121
2 votes
Accepted

Saving Large Spark ML Pipeline to HDFS

You can do the following: A Pipeline can be made of other pipelines. Isn't that great? A Pipeline inherit from the Estimator class and by definition, a PipelineStage can be either an Estimator or a ...
Felipe Bormann's user avatar
2 votes
Accepted

does storing file in hdfs parallelize it for Spark?

As you can see for the following examples presented here at the documentation, you'd be able to read directly from HDFS without much trouble. And spark will parallelize the data for you properly. We ...
Felipe Bormann's user avatar
2 votes
Accepted

Are Hadoop and Python SciPy used for the same?

I think you're quite confused. Hadoop is a collection of software that contains a a distributed file system called HDFS. Essentially HDFS is a way to store data cross a cluster. You can access ...
Tophat's user avatar
  • 2,430
2 votes

BERT in production

You can just apply it with Spark. There is no reason you can't use Pytorch in a Spark job; just add it as a dependency when you submit the job. Spark's pandas UDFs can be pretty useful for scoring ...
Sean Owen's user avatar
  • 6,595
1 vote

Predictive Analytics on distributed systems vs standalone system

There is nothing special about building something across many machines, or Hadoop, so maybe we can take that out of the question. One general reason is that, even if the model building process were ...
Sean Owen's user avatar
  • 6,595
1 vote
Accepted

Which one of these tasks will benefit the most from SPARK?

I think second job will benefit more from spark than the first one. The reason is machine learning and predictive models often run multiple iterations on data. As you have mentioned, spark is able to ...
Theudbald's user avatar
  • 1,068
1 vote

Yarn service parameter for pseudodistributed

The mapreduce_shuffle in this config file is part of Plugable Shuffle and Sort. Shuffle and Sort are what connect the mappers to the reducers. A nice graphical ...
Stephen Rauch's user avatar
  • 1,783
1 vote
Accepted

Hadoop and input informations divided in splits

All bigdata eco system works on something called parallel processing. We have to process 100gigs of file. If we didnt split the file, then all the 100 gigs should be processed by single JVM(single ...
loneStar's user avatar
  • 126
1 vote
Accepted

Install Spark and Hadoop in the same machine

The Hadoop stack is difficult to setup and people complain that you can't trust any answers to problems over 6-12 months old. I would recommend getting a pre-configured Hadoop/Spark setup from ...
CalZ's user avatar
  • 1,663
1 vote

what ETL technique should i use for text documents using Hadoop?

I would suggest a data fabric. That would meet your need for data acquisition, preprocessing, data quality, master data management, etc. Given I work for Talend, I would suggest our data fabric. =) ...
Christopher Klaus's user avatar
1 vote
Accepted

Can R + Hadoop overcome R's memory constraints in any case?

R+Hadoop themselves do not actually give you any massive direct benefit. You could use Hadoop streaming to run parallel R jobs across all the nodes on your Hadoop cluster, but that is reliant on your ...
Henry's user avatar
  • 186
1 vote

Can R + Hadoop overcome R's memory constraints in any case?

To answer your question. I would give an analogous answer to clarify it better. Apache Spark is also one of the most prominent big data tools in the market. It too uses an In-Memory computation to run ...
user-116's user avatar
  • 671
1 vote

How many people can use a single Hadoop cluster at one time?

The bottleneck depends on the use pattern rather than the direct number of users. If people are doing high I/O workloads, then you wont get many people on at all. Whereas if you are doing small ...
Henry's user avatar
  • 186
1 vote

How many people can use a single Hadoop cluster at one time?

For initial learning you could very easily do proof of concept work against individual VMs (a 4GB VM with a pseudo cluster is enough to do basic mapreduce examples in). If you're going to use Spark I ...
Terra Field's user avatar
1 vote

Mahout Spark shell not working

This error is common when the SPARK_HOME environment variable is not set. In the shell type ...
rawkintrevo's user avatar
1 vote

How to Scaling Out Artifical Neural Networks?

You might take a look over TensorFrames, a Databricks library which allow running TensorFlow code on top of Apache Spark : https://github.com/databricks/tensorframes
Morgan Funtowicz's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible