Questions tagged [apache-spark]
Apache Spark is an open source cluster computing system that aims to make data analytics fast — both fast to run and fast to write, originally developed in the AMPLab at UC Berkeley.
242
questions
35
votes
6
answers
171k
views
Merging multiple data frames row-wise in PySpark
I have 10 data frames pyspark.sql.dataframe.DataFrame, obtained from randomSplit as ...
- 1,147
34
votes
5
answers
13k
views
What are the use cases for Apache Spark vs Hadoop
With Hadoop 2.0 and YARN Hadoop is supposedly no longer tied only map-reduce solutions. With that advancement, what are the use cases for Apache Spark vs Hadoop considering both sit atop of HDFS? I've ...
- 521
15
votes
3
answers
107k
views
How to calculate the mean of a dataframe column and find the top 10%
I am very new to Scala and Spark, and am working on some self-made exercises using baseball statistics. I am using a case class create a RDD and assign a schema to the data, and am then turning it ...
- 253
15
votes
3
answers
32k
views
How to convert categorical data to numerical data in Pyspark
I am using Ipython notebook to work with pyspark applications. I have a CSV file with lots of categorical columns to determine whether the income falls under or over the 50k range. I would like to ...
- 1,065
14
votes
3
answers
57k
views
Replace all numeric values in a pyspark dataframe by a constant value
Consider a pyspark dataframe consisting of 'null' elements and numeric elements. In general, the numeric elements have different values. How is it possible to replace all the numeric values of the ...
- 141
12
votes
3
answers
48k
views
How do I set/get heap size for Spark (via Python notebook)
I'm using Spark (1.5.1) from an IPython notebook on a macbook pro. After installing Spark and Anaconda, I start IPython from a terminal by executing: ...
- 303
12
votes
3
answers
6k
views
Issue with IPython/Jupyter on Spark (Unrecognized alias)
I am working on setting up a set of VMs to experiment with Spark before I spend go out and spend money on building up a cluster with some hardware. Quick note: I am an academic with a background in ...
- 121
12
votes
1
answer
4k
views
Spark ALS: recommending for new users
The question
How do I predict the rating for a new user in an ALS model trained in Spark? (New = not seen during training time)
The problem
I'm following the official Spark ALS tutorial here:
...
- 236
11
votes
3
answers
7k
views
When does cache get expired for a RDD in pyspark?
We use .cache() on RDD for persistent caching of an dataset, My concern is when this cached will be expired?.
...
- 1,147
11
votes
1
answer
22k
views
PySpark dataframe repartition
What happens when we do repartition on a PySpark dataframe based on the column. For example
dataframe.repartition('id')
Does this moves the data with the similar ...
- 213
11
votes
1
answer
21k
views
Calculate cosine similarity in Apache Spark
I have a DataFrame with IDF of certain words computed.
For example
...
- 243
10
votes
1
answer
17k
views
Server log analysis using machine learning
I was assigned this task to analyze the server logs of our application which contains exception logs, database logs event logs etc. I am new to machine learning, we use Spark with elastic search and ...
- 111
10
votes
1
answer
15k
views
Spark, optimally splitting a single RDD into two
I have a large dataset that I need to split into groups according to specific parameters. I want the job to process as efficiently as possible. I can envision two ways of doing so
Option 1 - Create ...
- 1,215
9
votes
3
answers
9k
views
How to run a pyspark application in windows 8 command prompt
I have a python script written with Spark Context and I want to run it. I tried to integrate IPython with Spark, but I could not do that. So, I tried to set the spark path [ Installation folder/bin ] ...
- 1,065
9
votes
1
answer
5k
views
Extracting individual emails from an email thread
Most of the open source datasets are well formatted i.e each email message is separated well like the enron email dataset. But out in the real world it is highly difficult to separate a top email ...
- 153
8
votes
4
answers
98k
views
How to select particular column in Spark(pyspark)?
testPassengerId = test.select('PassengerId').map(lambda x: x.PassengerId)
I want to select PassengerId column and make RDD of it. But .select is not working. It says 'RDD' object has no attribute '...
- 181
8
votes
2
answers
924
views
Understanding how distributed PCA works
As part of big data analysis project, I'm working on,
I need to perform PCA on some data, using cloud computing system.
In my case, I'm using Amazon EMR for the job and Spark in particular.
Leaving ...
- 183
8
votes
4
answers
6k
views
Unbalanced class: class_weight for ML algorithms in Spark MLLib
In python sklearn, there are multiple algorithms (e.g. regression, random forest ... etc.) that have the class_weight parameter to handle unbalanced data.
However, I do not find such parameter for ...
- 2,755
7
votes
1
answer
3k
views
Using Apache Spark to do ML. Keep getting serializing errors
so I'm using Spark to do sentiment analysis, and I keep getting errors with the serializers it uses (I think) to pass python objects around.
...
- 71
7
votes
1
answer
4k
views
Why does logistic regression in Spark and R return different models for the same data?
I've compared the logistic regression models on R (glm) and on Spark (LogisticRegressionWithLBFGS) on a dataset of 390 obs. of ...
- 113
7
votes
1
answer
2k
views
Why does spark.ml.feautures.Word2Vec vectorize sentences instead of single words?
In the process of understanding how Word2Vec in Spark differs from gensim one, I got very confused by the example presented in the Spark docs (reference link:
https://spark.apache.org/docs/2.2.0/ml-...
- 193
7
votes
2
answers
8k
views
SPARK Mllib: Multiclass logistic regression, how to get the probabilities of all classes rather than the top one?
I am using LogisticRegressionWithLBFGS to train a multi-class classifier.
Is there a way to get the probability of all classes (not only the top candidate class) ...
- 594
7
votes
1
answer
1k
views
SPARK, ML: Naive Bayes classifier often assigns 1 as probability prediction
Hi I am using Spark ML to optimise a Naive Bayes multi-class classifier.
I have about 300 categories and I am classifying text documents. The training set is balanced enough and there is about 300 ...
- 594
7
votes
1
answer
9k
views
How to implement LSTM with Spark?
I would like to build an LSTM network for text classification with PySpark, but I don't find any library or function about it.
...
- 715
7
votes
1
answer
4k
views
Item-Item similarity based on text
We're build an item-item recommender based on the text descriptions of the items. Our initial approach was to calculate the TF-IDF vectors for each item. We used a hashing tf with 5000 possible hashes ...
7
votes
1
answer
1k
views
Pyspark: Filter dataframe based on separate specific conditions
How can I select only certain entries that match my condition and from those entries, filter again using regex?
For instance, I have this data frame (df):
col1
col2
col3
col4
A
f
5
g
D
er
2e
sd
F
...
7
votes
0
answers
942
views
ALS in Spark: what loss function is it minimizing?
I’ve playing with the MovieLens ratings dataset under Spark’s ALS and a manual implementation of ALS and comparing results with the same hyperparameters. I’d like to know this exactly in order to make ...
- 874
6
votes
3
answers
19k
views
Reading CSVs with new lines in fields with Spark
I was trying to load the below weblogic domain log (application error log) into Spark dataframe. I created a RDD and converted the RDD into dataframe. I was able to load the data successfully for the ...
- 61
6
votes
2
answers
5k
views
What are the alternatives to Python + Spark (pyspark)?
I like Python, and I like Spark, but they don't fit very well together. In particular,
it is very hard to use python functions in spark (have to create JVM binding for function in python)
it is hard ...
- 163
5
votes
2
answers
3k
views
Distributed k-means in Spark
I want to implement K-means algorithm in Spark. I am looking for a starting point and I found Berkeley's naive implementation. However, is that distributed?
I mean I see no mapreduce operations. Or ...
- 291
5
votes
1
answer
899
views
Machine Learning in Spark
I am using Apache Spark to perform sentiment analysis.I am using Naive Bayes algorithm to classify the text. I don't know how to find out the probability of labels. I would be grateful if I know get ...
5
votes
2
answers
1k
views
Local Development for Apache Spark
I'm wondering how other developers are setting up their local environments for working on Spark projects. Do you configure a 'local' cluster using a tool like Vagrant? Or, is it most common to SSH ...
- 1,169
5
votes
1
answer
2k
views
Saving Large Spark ML Pipeline to HDFS
I'm having trouble saving a large (relative to spark.rpc.message.maxSize) Spark ML pipeline to HDFS. Specifically, when I try to save the model to HDFS, it gives me an error related to spark's maximum ...
- 1,525
5
votes
2
answers
954
views
Can theano work on mapreduce or on spark?
I am not sure whether the Theano library can be used to write parallelized code in map reduce or in spark. Any expert opinion is welcome. A discussion was on at:
Theano-dev
- 571
5
votes
1
answer
600
views
Using Spark for finding similar users to a user?
I read about
https://mahout.apache.org/users/algorithms/intro-cooccurrence-spark.html
but couldn't find a spark library for this implementation.
I have columnar string dataset.
I have a dataset ...
- 191
5
votes
2
answers
293
views
Random Forest Regression. How to represent really long list of categories for processing
I'm trying to build a model to solve a regression task.
Simplified, the data look like:
...
- 151
4
votes
2
answers
2k
views
How to start prediction from dataset?
I have event dataset in a factless table. It has list of events
timestamp-> event name -> node ( In network)
There is always a sequence of event happening. ...
- 93
4
votes
3
answers
12k
views
Remove all columns where the entire column is null
I have a very dirty csv where there are several columns with only null values.
I would like to remove them. I am trying to select all columns where the count of null values in the column is not equal ...
- 143
4
votes
1
answer
16k
views
How to select multiple columns in a RDD with Spark (pySpark)?
Lets say I have a RDD that has comma delimited data. Each comma delimited value represents the amount of hours slept in the day of a week.
So for i.e. [8,7,6,7,8,8,5]
How can I manipulate the RDD so ...
- 143
4
votes
2
answers
2k
views
Choosing between Storm+Trident-ML, Storm+SAMOA or Spark Streaming+MLlib
I want to implement Streaming Naive Bayes in a distributed system. What are the best approach to choose framework. Should I choose:
Storm alone and implement streaming naive bayes on my own in storm ...
- 141
4
votes
2
answers
5k
views
How Mllib in Spark select variables in logistic regression
I have a question about MLlib in Spark.(with Scala)
I'm trying to understand how LogisticRegressionWithLBFGS and LogisticRegressionWithSGD work. I usually use SAS or R to do logistic regressions but ...
- 113
4
votes
1
answer
838
views
What is the main difference between Hadoop and Spark? [closed]
I recently read the following about Hadoop vs. Spark:
Insist upon in-memory columnar data querying. This was the killer-feature that let Apache Spark run in seconds the queries that would take Hadoop ...
- 41
4
votes
1
answer
404
views
Is there any point in learning Hadoop in 2018?
I'm a data analyst/scientist working mostly with the Python open source stack i.e. Pandas, scikit-learn, matplotlib, seaborn etc. I want to expand my toolbox and learn a distributed computing ...
- 141
4
votes
1
answer
586
views
Scan-based operations Apache Spark
Looking at the first paper on RDDs/Apache Spark, I found a statement saying that "RDDs degrade gracefully when there is not enough memory to store them, as long as they are only being used in scan-...
- 347
4
votes
1
answer
1k
views
Has anyone succeeded in finding a good Scala/Spark kernel for Jupyter?
The ones I've tried so far
Almond: Works very well for just Scala, but you have to import dependencies, and it gets tedious after a while. And unfortunately can't run when using Spark with YARN ...
- 141
4
votes
2
answers
270
views
Lightweight execution of Spark MLLib models
I have some training data which I am using to build a Spark MLLib model which is in a Hive database. I am using simple linear regression models and the PySpark API.
I have a code set up to train this ...
- 141
4
votes
1
answer
3k
views
Spark MLLib - how to re-use TF-IDF model
I am using spark ml IDF estimator/model (TF-IDF) to convert text features into vectors before passing it to the classification algorithm.
Here's the process:
Datasets:
...
- 237
4
votes
0
answers
1k
views
Performance profiling and tuning in Apache Spark [closed]
Beyond the immediate suspects defined in the spark documentation, what are some ways to profile, tune and boost performance of an Apache Spark application?
- 141
3
votes
2
answers
497
views
Clustering a very large number of very small clusters with most data unrelated
I'm trying to detect duplicates in a data set of about 34k distinct items. When I say "duplicate," I don't mean identical items, just very similar. I have an algorithm that will Cartesian join the ...
- 141
3
votes
3
answers
2k
views
