Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [pyspark]

The tag has no usage guidance.

0
votes
0answers
7 views

Why does Spark persist to disk only lock executors? [closed]

I would assume .persist(StorageLevel.DISK_ONLY) would free up executor CPU space for coworkers. I.e. several different spark contexts performing work when needed, and storing intermediate RDDs to ...
0
votes
0answers
14 views

How to find number of Gaussians in GaussianMixture?

I have been using the elbow method to find optimal K value in K-means. Now while using GaussianMixture (Using Spark), how to find the number of gaussians say 2 OR 3 OR more...? Thanks
0
votes
1answer
21 views

Reliable way to verify Pyspark data frame column type

If I read data from a CSV, all the columns will be of "String" type by default. Generally, I inspect the data using the following functions which gives an overview of the data and its types ...
0
votes
1answer
99 views

Replacing null with average in pyspark

I have a problem during upsampling operation in PySpark. My dataframe is: ...
0
votes
1answer
62 views

Plotting in PySpark?

I have been searching for methods to plot in PySpark. I couldn't find any resource on plotting data residing in DataFrame in PySpark. The only methods which are listed are: through method ...
0
votes
0answers
30 views

How to apply class imbalance techniques on a pyspark.sql.dataframe.DataFrame

I am working on a binary classification model(using RF, logit, and svm) and my data is highly imbalanced, so I am trying to use 'SMOTE' etc techniques for oversampling the minority class. Currently, ...
0
votes
0answers
9 views

Introspection of lambda function within Spark RDD function

I have the following code: ...
0
votes
1answer
21 views

Apache Spark Understanding

I have read a lot about Apache Spark. About RDD, DataFrames etc etc. But I haven't come across a simple explanation as to why and where should we be using Apache Spark? If my primary goal is machine/...
0
votes
0answers
14 views

Error while accessing prediction results on Zeppelin

I trained a RandomForestClassifier(rfModel) on Zeppelin and trying to test it using raw data. I am running into error when I try to access the predicted dataframe using show()/print functions. Any ...
1
vote
0answers
88 views

Frequent itemset vs. frequent sequence mining algorithms on variables with an inherent order

I've been using Pyspark's PrefixSpan algorithm to mine sequences of variables with an inherent order (bus stops). I know that PrefixSpan takes into consideration the order of the items it is fed, ...
0
votes
0answers
33 views

Streaming Recommender System using Spark

I have trained a recommender model using ALS from spark.mlib. Now I want to consider the incoming ratings to my model. My data is in form of spark's DataFrame. The idea is to use spark streaming, so I ...
3
votes
1answer
25 views

k-means|| in PySpark

I'm trying to apply k-means$\|$ clustering in PySpark. According to this paper, there is an oversampling factor, $l$, that would affect the model's cost. I couldn't find any parameter regarding ...
1
vote
0answers
2k views

Adding multiple columns to spark dataframe

I would like to add several columns to a spark (actually pyspark) dataframe , these columns all being functions of several input columns in the df. There seems to be no 'add_columns' in spark, and ...
1
vote
0answers
50 views

Running H2O in databricks

I'm trying to run H2O in databricks. However, when I do the following: hc = pysparkling.H2OContext.getOrCreate(spark) I get the following error: java.lang.AbstractMethodError Does anyone know what ...
0
votes
2answers
69 views

Exponential runtime on distance matrix

I have an exploratory script running a Databricks notebook that performs a simple arithmetic function (Pythagorean theorem) on all possible pairwise combinations of a list of pairs of floats (akin to ...
3
votes
1answer
753 views
2
votes
1answer
24 views

PySpark Filter shows only 1 row

I have a 12 GB .csv file and I am trying to filter rows based on some value in a column under "AppName" Doing this in Pyspark: ...
2
votes
1answer
334 views

What are the alternatives to Python + Spark (pyspark)?

I like Python, and I like Spark, but they don't fit very well together. In particular, it is very hard to use python functions in spark (have to create JVM binding for function in python) it is ...
0
votes
1answer
185 views

Use of Random Forest algorithm in PySpark for imputation

I am wondering how to use Random Forest algorithm for imputing missing values in a dataset. It is supposed to work well with missing values but I am not sure how those missing values are dealt with ...
2
votes
1answer
778 views

Run python script using spark-submit on windows 7

I'm trying to submit a python script using spark-submit on windows 7. I'm using spark 2.3.0 built for hadoop 2.7 . The python script is supposed to read data from a csv file and plot graphs. is it ...
0
votes
0answers
15 views

Recommender system: Give a feature more significance than another

I am trying to build a recommender system that predicts hotel prices based on a great number of features. I have a column representing the hotel rating out of 5 and another column indicating the ...
0
votes
0answers
86 views
3
votes
1answer
2k views

PySpark dataframe repartition

What happens when we do repartition on a PySpark dataframe based on the column. For example dataframe.repartition('id') Does this moves the data with the similar ...
0
votes
2answers
47 views

KMeans for crime patterns

I'm trying to find the patterns in the crime records I have in a database. I thought clustering would be a way to do it. This is my (cooked up) dataset: ...
0
votes
1answer
620 views

Filling missing values with pyspark using a probability distribution

I want to fill missing values in my dataframe. ...
0
votes
0answers
711 views

Similarity Matrix in PySpark

I want to generate similarity matrix (based on euclidean distance) of a large dataset on a spark cluster in pyspark. How can it be done??
1
vote
1answer
5k views

Is there any way to read Xlsx file in pyspark?Also want to read strings of column from each columnName

pd is a panda module is one way of reading excel but its not available in my cluster. I want to read excel without pd module. Code1 and Code2 are two implementations i want in pyspark. Code 1: ...
0
votes
1answer
61 views

Install Spark and Hadoop in the same machine [closed]

I'm trying to install Spark Apache and Hadoop in the same machine. Spark will be used to process data, and HDFS from Hadoop will be used to store data. I first started by installing spark and it ...
4
votes
1answer
988 views

Spark development on local machine with PyCharm

I am wondering what is the best practice other devs are using for their python spark jobs. I am building a dev environment in which I am looking to write code in PyCharm with SparkContext pointing to ...
1
vote
0answers
105 views

Creating a user-based recommendation engine?

I have a data set of some questions and answers that users have completed by choices. I'm trying to build a recommendation engine to find similar users based on their answers on top of this. So for ...
1
vote
0answers
363 views

How to do Feature Extraction using Apache Spark

I am a newbie to Machine Learning and I have to do feature extraction for a banking application to detect / predict fraudulent transactions. I have come across few articles on Feature Extraction ...
-1
votes
1answer
613 views

Words from LDA output pyspark ml

I have built a pipelined lda model using pyspark's ml. The code is below: tokenizer = Tokenizer(inputCol='formatted_transcript', outputCol='words') stopWordsRemover = StopWordsRemover(...
1
vote
1answer
3k views

Plot RDD data using a pyspark dataframe from csv file [closed]

I have just started working with pyspark on very large csv file. I am using Spark version 2.1.0. I want to read data from a .csv file and load it into a spark dataframe and then after filtering ...
0
votes
1answer
302 views

Pyspark code is not performant enough when compared to pure python alternative

I transformed the existing code which was in python pasted below was in pyspark. Python code: ...
0
votes
2answers
48 views

Predicting the variable's score using predictive analytics

I am new to predictive analytics but I am good at programming (like Spark, R). I have 10 variables and I know their score's for over a long period of time. I want to know which variable will have ...
1
vote
0answers
73 views

Predicting sales in unobserved locations

I am building a model that is trying to predict the 5 year sales production of a salesperson in every zip code in the United States. I have only 7 years of data. This task is one piece of a larger ...
2
votes
2answers
1k views

Is there any use to running Pandas on Spark?

I just started working with PySpark this week, and the instance I have access two has Pandas installed. But what use is having Pandas on Spark? Isn't the whole purpose of running scripts on PySpark ...
1
vote
1answer
3k views

Dataframe request with groupBy

I'm a beginner in Spark and I want to calculate the average of number per name. I have a JSON file with this information ...
4
votes
0answers
66 views

RDD of gziped files to “uncompressed” Dataframe [closed]

I have a bunch of compressed text files each line of which containing a JSON object. Simplified my workflow looks like this: ...
1
vote
2answers
3k views

spark item similarity recommendation

I would like to build an recommendation engine using spark's Mlib itemsimilarity as mentioned here LINK But it seems spark do not have this algorithm any more and ...
10
votes
1answer
2k views

Spark ALS: recommending for new users

The question How do I predict the rating for a new user in an ALS model trained in Spark? (New = not seen during training time) The problem I'm following the official Spark ALS tutorial here: ...
1
vote
1answer
511 views

PrefixSpan model input RDD format

I am trying to understand the input for the PrefixSpan model (pyspark). I thought in the input RDD, each row should be a list. However, as I looked into the source code, each row is actually a list of ...
0
votes
1answer
2k views

Correct way to set Spark variables in jupyter notebook

I need to set a couple of variables in my Jupyter Notebook where I have a pre-existing sparkContext and sqlContext and I am doing it wrong. If I don't include an sc.stop() , I get an error that I ...
2
votes
1answer
2k views

Logistic Regression as multiclass classification using PySpark and issues

I am trying to use Logistic Regression to classify the datasets which has Sparse Vector in feature vector: Case 1: I tried using the pipeline of ML in MLLIB as ...
1
vote
1answer
6k views

Converting RDD to spark data frames in python and then accessing a particular values of columns

I want to access values of a particular column from a data sets that I've read from a csv file. The datasets are stored in pyspark RDD which I want to be converted into the DataFrame. I am using the ...
8
votes
3answers
39k views

Import csv file contents into pyspark dataframes

How can I import a .csv file into pyspark dataframes ? I even tried to read csv file in Pandas and then converting it to spark dataframes using createDataFrame but it is still showing some error. Can ...
4
votes
1answer
512 views

Machine Learning in Spark

I am using Apache Spark to perform sentiment analysis.I am using Naive Bayes algorithm to classify the text. I don't know how to find out the probability of labels. I would be grateful if I know get ...
3
votes
3answers
5k views

Pyspark coverting timestamps from UTC to many timezones

This is using python with Spark 1.6.1 and dataframes. I have timestamps in UTC that I want to convert to local time, but a given row could be in any of several timezones. I have an 'offset' value (or ...
1
vote
2answers
164 views

Spark DataFrame courses

If I understand the Databricks philosophy correctly, Spark will soon be heavily moving toward dataframes, i.e. away from the usual map/reduce on RDDs. I was wondering if there are any good suggestions ...
2
votes
1answer
2k views

Unable to load NLTK in spark using PySpark

I have install NLTK and its working fine with the following code, I running in pyspark shell ...