Skip to main content

Questions tagged [apache-spark]

Apache Spark is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write, originally developed in the AMPLab at UC Berkeley.

Filter by
Sorted by
Tagged with
35 votes
6 answers
171k views

Merging multiple data frames row-wise in PySpark

I have 10 data frames pyspark.sql.dataframe.DataFrame, obtained from randomSplit as ...
krishna Prasad's user avatar
34 votes
5 answers
13k views

What are the use cases for Apache Spark vs Hadoop

With Hadoop 2.0 and YARN Hadoop is supposedly no longer tied only map-reduce solutions. With that advancement, what are the use cases for Apache Spark vs Hadoop considering both sit atop of HDFS? I've ...
idclark's user avatar
  • 521
15 votes
3 answers
107k views

How to calculate the mean of a dataframe column and find the top 10%

I am very new to Scala and Spark, and am working on some self-made exercises using baseball statistics. I am using a case class create a RDD and assign a schema to the data, and am then turning it ...
the3rdNotch's user avatar
15 votes
3 answers
32k views

How to convert categorical data to numerical data in Pyspark

I am using Ipython notebook to work with pyspark applications. I have a CSV file with lots of categorical columns to determine whether the income falls under or over the 50k range. I would like to ...
SRS's user avatar
  • 1,065
14 votes
3 answers
57k views

Replace all numeric values in a pyspark dataframe by a constant value

Consider a pyspark dataframe consisting of 'null' elements and numeric elements. In general, the numeric elements have different values. How is it possible to replace all the numeric values of the ...
justus's user avatar
  • 141
12 votes
3 answers
48k views

How do I set/get heap size for Spark (via Python notebook)

I'm using Spark (1.5.1) from an IPython notebook on a macbook pro. After installing Spark and Anaconda, I start IPython from a terminal by executing: ...
Kai's user avatar
  • 303
12 votes
3 answers
6k views

Issue with IPython/Jupyter on Spark (Unrecognized alias)

I am working on setting up a set of VMs to experiment with Spark before I spend go out and spend money on building up a cluster with some hardware. Quick note: I am an academic with a background in ...
gcd's user avatar
  • 121
12 votes
1 answer
4k views

Spark ALS: recommending for new users

The question How do I predict the rating for a new user in an ALS model trained in Spark? (New = not seen during training time) The problem I'm following the official Spark ALS tutorial here: ...
ciri's user avatar
  • 236
11 votes
3 answers
7k views

When does cache get expired for a RDD in pyspark?

We use .cache() on RDD for persistent caching of an dataset, My concern is when this cached will be expired?. ...
krishna Prasad's user avatar
11 votes
1 answer
22k views

PySpark dataframe repartition

What happens when we do repartition on a PySpark dataframe based on the column. For example dataframe.repartition('id') Does this moves the data with the similar ...
Nikhil Baby's user avatar
11 votes
1 answer
21k views

Calculate cosine similarity in Apache Spark

I have a DataFrame with IDF of certain words computed. For example ...
Ganesh Krishnan's user avatar
10 votes
1 answer
17k views

Server log analysis using machine learning

I was assigned this task to analyze the server logs of our application which contains exception logs, database logs event logs etc. I am new to machine learning, we use Spark with elastic search and ...
elric's user avatar
  • 111
10 votes
1 answer
15k views

Spark, optimally splitting a single RDD into two

I have a large dataset that I need to split into groups according to specific parameters. I want the job to process as efficiently as possible. I can envision two ways of doing so Option 1 - Create ...
j.a.gartner's user avatar
  • 1,215
9 votes
3 answers
9k views

How to run a pyspark application in windows 8 command prompt

I have a python script written with Spark Context and I want to run it. I tried to integrate IPython with Spark, but I could not do that. So, I tried to set the spark path [ Installation folder/bin ] ...
SRS's user avatar
  • 1,065
9 votes
1 answer
5k views

Extracting individual emails from an email thread

Most of the open source datasets are well formatted i.e each email message is separated well like the enron email dataset. But out in the real world it is highly difficult to separate a top email ...
Greedy Coder's user avatar
8 votes
4 answers
98k views

How to select particular column in Spark(pyspark)?

testPassengerId = test.select('PassengerId').map(lambda x: x.PassengerId) I want to select PassengerId column and make RDD of it. But .select is not working. It says 'RDD' object has no attribute '...
dsl1990's user avatar
  • 181
8 votes
2 answers
924 views

Understanding how distributed PCA works

As part of big data analysis project, I'm working on, I need to perform PCA on some data, using cloud computing system. In my case, I'm using Amazon EMR for the job and Spark in particular. Leaving ...
Adiel's user avatar
  • 183
8 votes
4 answers
6k views

Unbalanced class: class_weight for ML algorithms in Spark MLLib

In python sklearn, there are multiple algorithms (e.g. regression, random forest ... etc.) that have the class_weight parameter to handle unbalanced data. However, I do not find such parameter for ...
Edamame's user avatar
  • 2,755
7 votes
1 answer
3k views

Using Apache Spark to do ML. Keep getting serializing errors

so I'm using Spark to do sentiment analysis, and I keep getting errors with the serializers it uses (I think) to pass python objects around. ...
seashark97's user avatar
7 votes
1 answer
4k views

Why does logistic regression in Spark and R return different models for the same data?

I've compared the logistic regression models on R (glm) and on Spark (LogisticRegressionWithLBFGS) on a dataset of 390 obs. of ...
SparkUser's user avatar
  • 113
7 votes
1 answer
2k views

Why does spark.ml.feautures.Word2Vec vectorize sentences instead of single words?

In the process of understanding how Word2Vec in Spark differs from gensim one, I got very confused by the example presented in the Spark docs (reference link: https://spark.apache.org/docs/2.2.0/ml-...
datapug's user avatar
  • 193
7 votes
2 answers
8k views

SPARK Mllib: Multiclass logistic regression, how to get the probabilities of all classes rather than the top one?

I am using LogisticRegressionWithLBFGS to train a multi-class classifier. Is there a way to get the probability of all classes (not only the top candidate class) ...
Rami's user avatar
  • 594
7 votes
1 answer
1k views

SPARK, ML: Naive Bayes classifier often assigns 1 as probability prediction

Hi I am using Spark ML to optimise a Naive Bayes multi-class classifier. I have about 300 categories and I am classifying text documents. The training set is balanced enough and there is about 300 ...
Rami's user avatar
  • 594
7 votes
1 answer
9k views

How to implement LSTM with Spark?

I would like to build an LSTM network for text classification with PySpark, but I don't find any library or function about it. ...
Simone's user avatar
  • 715
7 votes
1 answer
4k views

Item-Item similarity based on text

We're build an item-item recommender based on the text descriptions of the items. Our initial approach was to calculate the TF-IDF vectors for each item. We used a hashing tf with 5000 possible hashes ...
Brandon Hamric's user avatar
7 votes
1 answer
1k views

Pyspark: Filter dataframe based on separate specific conditions

How can I select only certain entries that match my condition and from those entries, filter again using regex? For instance, I have this data frame (df): col1 col2 col3 col4 A f 5 g D er 2e sd F ...
randomizer0000's user avatar
7 votes
0 answers
943 views

ALS in Spark: what loss function is it minimizing?

I’ve playing with the MovieLens ratings dataset under Spark’s ALS and a manual implementation of ALS and comparing results with the same hyperparameters. I’d like to know this exactly in order to make ...
anymous.asker's user avatar
6 votes
3 answers
19k views

Reading CSVs with new lines in fields with Spark

I was trying to load the below weblogic domain log (application error log) into Spark dataframe. I created a RDD and converted the RDD into dataframe. I was able to load the data successfully for the ...
uk2016's user avatar
  • 61
6 votes
2 answers
5k views

What are the alternatives to Python + Spark (pyspark)?

I like Python, and I like Spark, but they don't fit very well together. In particular, it is very hard to use python functions in spark (have to create JVM binding for function in python) it is hard ...
stackoverflower's user avatar
5 votes
2 answers
3k views

Distributed k-means in Spark

I want to implement K-means algorithm in Spark. I am looking for a starting point and I found Berkeley's naive implementation. However, is that distributed? I mean I see no mapreduce operations. Or ...
gsamaras's user avatar
  • 291
5 votes
1 answer
899 views

Machine Learning in Spark

I am using Apache Spark to perform sentiment analysis.I am using Naive Bayes algorithm to classify the text. I don't know how to find out the probability of labels. I would be grateful if I know get ...
Vignesh Mohan's user avatar
5 votes
2 answers
1k views

Local Development for Apache Spark

I'm wondering how other developers are setting up their local environments for working on Spark projects. Do you configure a 'local' cluster using a tool like Vagrant? Or, is it most common to SSH ...
sheldonkreger's user avatar
5 votes
1 answer
2k views

Saving Large Spark ML Pipeline to HDFS

I'm having trouble saving a large (relative to spark.rpc.message.maxSize) Spark ML pipeline to HDFS. Specifically, when I try to save the model to HDFS, it gives me an error related to spark's maximum ...
Thomas Cleberg's user avatar
5 votes
2 answers
954 views

Can theano work on mapreduce or on spark?

I am not sure whether the Theano library can be used to write parallelized code in map reduce or in spark. Any expert opinion is welcome. A discussion was on at: Theano-dev
0xF's user avatar
  • 571
5 votes
1 answer
600 views

Using Spark for finding similar users to a user?

I read about https://mahout.apache.org/users/algorithms/intro-cooccurrence-spark.html but couldn't find a spark library for this implementation. I have columnar string dataset. I have a dataset ...
Nikhil Verma's user avatar
5 votes
2 answers
293 views

Random Forest Regression. How to represent really long list of categories for processing

I'm trying to build a model to solve a regression task. Simplified, the data look like: ...
evgenii's user avatar
  • 151
4 votes
2 answers
2k views

How to start prediction from dataset?

I have event dataset in a factless table. It has list of events timestamp-> event name -> node ( In network) There is always a sequence of event happening. ...
kinkajou's user avatar
4 votes
3 answers
12k views

Remove all columns where the entire column is null

I have a very dirty csv where there are several columns with only null values. I would like to remove them. I am trying to select all columns where the count of null values in the column is not equal ...
CJL's user avatar
  • 143
4 votes
1 answer
16k views

How to select multiple columns in a RDD with Spark (pySpark)?

Lets say I have a RDD that has comma delimited data. Each comma delimited value represents the amount of hours slept in the day of a week. So for i.e. [8,7,6,7,8,8,5] How can I manipulate the RDD so ...
Kent Wong's user avatar
  • 143
4 votes
2 answers
2k views

Choosing between Storm+Trident-ML, Storm+SAMOA or Spark Streaming+MLlib

I want to implement Streaming Naive Bayes in a distributed system. What are the best approach to choose framework. Should I choose: Storm alone and implement streaming naive bayes on my own in storm ...
Raman's user avatar
  • 141
4 votes
2 answers
5k views

How Mllib in Spark select variables in logistic regression

I have a question about MLlib in Spark.(with Scala) I'm trying to understand how LogisticRegressionWithLBFGS and LogisticRegressionWithSGD work. I usually use SAS or R to do logistic regressions but ...
SparkUser's user avatar
  • 113
4 votes
1 answer
843 views

What is the main difference between Hadoop and Spark? [closed]

I recently read the following about Hadoop vs. Spark: Insist upon in-memory columnar data querying. This was the killer-feature that let Apache Spark run in seconds the queries that would take Hadoop ...
Ironclad's user avatar
4 votes
1 answer
404 views

Is there any point in learning Hadoop in 2018?

I'm a data analyst/scientist working mostly with the Python open source stack i.e. Pandas, scikit-learn, matplotlib, seaborn etc. I want to expand my toolbox and learn a distributed computing ...
RodrikTheReader's user avatar
4 votes
1 answer
586 views

Scan-based operations Apache Spark

Looking at the first paper on RDDs/Apache Spark, I found a statement saying that "RDDs degrade gracefully when there is not enough memory to store them, as long as they are only being used in scan-...
mbbce's user avatar
  • 347
4 votes
1 answer
1k views

Has anyone succeeded in finding a good Scala/Spark kernel for Jupyter?

The ones I've tried so far Almond: Works very well for just Scala, but you have to import dependencies, and it gets tedious after a while. And unfortunately can't run when using Spark with YARN ...
Varun Gawande's user avatar
4 votes
2 answers
271 views

Lightweight execution of Spark MLLib models

I have some training data which I am using to build a Spark MLLib model which is in a Hive database. I am using simple linear regression models and the PySpark API. I have a code set up to train this ...
Tom's user avatar
  • 141
4 votes
1 answer
3k views

Spark MLLib - how to re-use TF-IDF model

I am using spark ml IDF estimator/model (TF-IDF) to convert text features into vectors before passing it to the classification algorithm. Here's the process: Datasets: ...
nir's user avatar
  • 237
4 votes
0 answers
1k views

Performance profiling and tuning in Apache Spark [closed]

Beyond the immediate suspects defined in the spark documentation, what are some ways to profile, tune and boost performance of an Apache Spark application?
Eran Medan's user avatar
3 votes
2 answers
497 views

Clustering a very large number of very small clusters with most data unrelated

I'm trying to detect duplicates in a data set of about 34k distinct items. When I say "duplicate," I don't mean identical items, just very similar. I have an algorithm that will Cartesian join the ...
Tim Gautier's user avatar
3 votes
3 answers
2k views

Task not serializable Error

...
Credosam's user avatar

1
2 3 4 5