59 votes
Accepted

Merging multiple data frames row-wise in PySpark

Stolen from: https://stackoverflow.com/questions/33743978/spark-union-of-multiple-rdds Outside of chaining unions this is the only way to do it for DataFrames. ...
Jan van der Vegt's user avatar
20 votes

How do I set/get heap size for Spark (via Python notebook)

You can manage Spark memory limits programmatically (by the API). As SparkContext is already available in your Notebook: sc._conf.get('spark.driver.memory') ...
noleto's user avatar
  • 301
17 votes

Merging multiple data frames row-wise in PySpark

Sometime, when the dataframes to combine do not have the same order of columns, it is better to df2.select(df1.columns) in order to ensure both df have the same ...
Wong Tat Yau's user avatar
16 votes

How to convert categorical data to numerical data in Pyspark

This can be done using StringIndexer in PySpark and the reverse using IndexToString for reference please check this: ...
krishna Prasad's user avatar
13 votes
Accepted

When does cache get expired for a RDD in pyspark?

It will not expire until Spark is out of memory, at which point it will remove RDDs from cache which are used least often. When you ask for something that has been uncached it will recalculate the ...
Jan van der Vegt's user avatar
12 votes
Accepted

PySpark dataframe repartition

The default value for spark.sql.shuffle.partitions is 200, and configures the number of partitions that are used when shuffling data for joins or aggregations. <...
Kiran's user avatar
  • 136
10 votes

Spark ALS: recommending for new users

Lots of questions here. First, for a truly new user with no data, there is no way to use a recommender model. If you have literally no information on the user, the only thing you can do is provide ...
Sean Owen's user avatar
  • 6,595
9 votes

Replace all numeric values in a pyspark dataframe by a constant value

Using lit would convert all values of the column to the given value. To do it only for non-null values of dataframe, you would have to filter non-null values of ...
Santoshi M's user avatar
8 votes
Accepted

Calculate cosine similarity in Apache Spark

There's a related example to your problem in the Spark repo here. The strategy is to represent the documents as a RowMatrix and then use its columnSimilarities() method. That will get you a matrix ...
Pete's user avatar
  • 809
8 votes

Replace all numeric values in a pyspark dataframe by a constant value

As per your problem, I think it might be easier to use lit. Try this- from pyspark.sql.functions import lit new_df = df.withColumn('column_name', lit(1)) Hope it ...
Abhishek's user avatar
  • 1,959
7 votes

How to calculate the mean of a dataframe column and find the top 10%

This also returns the average of the selected column: ...
Erkan ŞİRİN's user avatar
7 votes
Accepted

Machine Learning in Spark

Probability can be found for the test dataset once you trained the model and transformed for the test dataset e.g: if your trained Naive Bayes model is model then ...
krishna Prasad's user avatar
7 votes

spark item similarity recommendation

For your recommendation engine, if you've chosen to go by item similarity approach, then you can use Spark's RowMatrix datatype to achieve this task. Item similarity approach is just about creating ...
Santoshi M's user avatar
6 votes
Accepted

Distributed k-means in Spark

In that link you posted, you can look at the python full solution here at the end and go through it to see what all is distributed. In short, some parts are distributed, like reading data from the ...
Harsh's user avatar
  • 1,051
6 votes

When does cache get expired for a RDD in pyspark?

In addition to Jan's answer, I would like to point out that serialized RDD storage(/caching) works much better than normal RDD caching for large datasets. It also helps optimize garbage collection, ...
Dawny33's user avatar
  • 8,296
6 votes
Accepted

How to start prediction from dataset?

The problem you are facing is a time series problem. Your events are categorial which is a specific case (so most common techniques like arima and Fourier transform are irrelevant). Before getting ...
DaL's user avatar
  • 2,633
6 votes
Accepted

Do categorical features always need to be encoded?

You have partly answered this question yourself ("because converting to integers implies that there is an ordering between features"). I will just clarify the terminology a bit more. Categorical ...
hssay's user avatar
  • 1,998
6 votes
Accepted

How to select multiple columns in a RDD with Spark (pySpark)?

I dont know which version you are using but I recommend DataFrames since most of upgrades are coming for DataFrames. (I prefer ...
Ilker Kurtulus's user avatar
5 votes

Reading CSVs with new lines in fields with Spark

There is a recently-added feature in Spark 2.2.0... spark.read.csv(file, multiLine=True) https://issues.apache.org/jira/browse/SPARK-19610 https://issues.apache....
Aneel's user avatar
  • 151
5 votes
Accepted

Understanding how distributed PCA works

The question is more related to Apache Spark architecture and map reduce; there are more than one questions here, however, the central piece of your question perhaps is For example, one of the means ...
Ironluca's user avatar
  • 187
5 votes
Accepted

Use cases for graph algorithms and graph data structures in finance and banking

There are many use cases of graph theory in Finance industry and it is a very broad question. As Emre said can be used for Fraud Detection, Risk Modelling, Economic Networks etc. These below links ...
Toros91's user avatar
  • 2,392
5 votes
Accepted

Why does spark.ml.feautures.Word2Vec vectorize sentences instead of single words?

Spark (naively) uses average of vectors for all words in the document as representation of the document. Check the API documentation little carefully. "The Word2VecModel transforms each document into ...
hssay's user avatar
  • 1,998
5 votes

Remove all columns where the entire column is null

The accepted answer will work, but will run df.count() for each column, which is quite taxing for a large number of columns. Calculate it once before the list ...
A F's user avatar
  • 51
4 votes

Merging multiple data frames row-wise in PySpark

How about using recursion? ...
proinsias's user avatar
  • 141
4 votes

SPARK Mllib: Multiclass logistic regression, how to get the probabilities of all classes rather than the top one?

I am working on Random Forest Classifier and this classifier has probability attribute in prediction i.e if you get the summary of ...
krishna Prasad's user avatar
4 votes

How do I set/get heap size for Spark (via Python notebook)

Just use the config option when setting SparkSession (as of 2.4) ...
LaSul's user avatar
  • 471
4 votes

How to select particular column in Spark(pyspark)?

You could try the following, testPassengerID = test.select('PassengerID').rdd this would select the column PassengerID and ...
user25409's user avatar
4 votes

Hashing trick with random forest in scala

Let's look at the error message: found : Array[org.apache.spark.mllib.linalg.Vector] required: org.apache.spark.mllib.linalg.Vector "found" is the type of object ...
Matthew Gray's user avatar
4 votes
Accepted

Apache Spark ML vs Flink ML

Both Spark and Flink are designed to process data in batch or stream over distributed environment. Flink primarily being defined as its ability to process streaming data in real time and being ...
Abhishek's user avatar
  • 1,959
4 votes
Accepted

Plotting in PySpark?

No, there is no such method, I have found out. The reason is, plotting libraries run on a single machine and expect a rather sample dataset. Data on Spark is distributed among its clusters and hence ...
Osama Dar's user avatar
  • 599

Only top scored, non community-wiki answers of a minimum length are eligible