Questions tagged [pyspark]
The Spark Python API (PySpark) exposes the apache-spark programming model to Python.
0
votes
0answers
8 views
how to generate a visualization showing the distribution of drug-abuse-related tweets throughout the USA? [closed]
I need to spatially join tweets that contains drug-related terms with the census tracts, and compute the normalized number of tweets using the provided boundary and population data.
I must implement ...
4
votes
1answer
93 views
Model ensemble with Spark or Scikit Learn
I am using Spark MLLib to make prediction and I would like to know if it is possible to create your custom Estimators.
Here is a reproducible of what I would like my model to do with the Spark api
<...
0
votes
0answers
4 views
Getting parameters of the best model with crossvalidation in with SparkMLLib
I am having trouble accessing the parameters of estimators of model in SparkMLlib. More precisely my problem is:
I have a logistic regression model for which I want to find the best regularization ...
0
votes
1answer
32 views
K-modes implementation in pyspark
I'm looking for an implementation of k-modes in pyspark. I found this and this as implementations.
First, I tried implementing k-modes using the first link and faced issues. So I went ahead and tried ...
2
votes
0answers
18 views
Obtain learning curve of Gradient Boosted Tree model in PySpark
Currently there seems to be no method in PySpark of checkpointing the performance of a model at each gradient update. Is there a way to get the performance of a model at each gradient update so that a ...
0
votes
0answers
8 views
Null predictions for ALS in Pyspark
I am trying to read from my dataset which has three coloumns. (User, Repository and Number of Stars)
In[10]
...
0
votes
0answers
16 views
Can we use gunicorn to deploy pyspark models?
I have created a pyspark model and want to deploy it as a real time application. I am using Flask to handle requests and gunicorn to handle multiple requests by increasing workers and threads. To run ...
2
votes
0answers
82 views
1
vote
2answers
76 views
PySpark v Pandas Dataframe Memory Issue
Suppose I have a csv file with 20k rows, when I import in Pandas dataframe format and run the ML algos like Random Forest or Logistic Regression from sklearn package it just runs fine.
But same thing ...
0
votes
0answers
28 views
What is the difference between PySpark's featuresCol, labelCol, predictionCol, and probabilityCol?
I am attempting to train a random forest classifier (pyspark.ml.classification.RandomForestClassifier) on a large dataset (~70gb). However, I am not sure what to ...
0
votes
1answer
98 views
Using keras tokenizer on Spark DataFrame. Column is not iterable [closed]
I am having trouble using the Tokenizer
The code is
...
1
vote
0answers
103 views
alternatives to pyspark to query a csv file locally (no distributed computation needed)
I am reading a csv file with pyspark to extract some information from it. I am running pyspark locally and I do not need distributed computation. On the contrary I need some plotting library such as ...
1
vote
0answers
12 views
Is there a built-in pyspark function for the paradigm union(filter, take)? [closed]
Essentially, filtering by key, taking a specific number (based on key) from the DataFrame, and unioning these results for a number of values.
For example,
...
1
vote
1answer
39 views
Best Data Science & Machine Learning Conferences 2019? [closed]
What are the best data science conferences you have attended? I am looking for a conference that will help me further develop my Data Science and Machine Learning competencies. Topics that are ...
0
votes
1answer
57 views
Why is there a difference of “ML” vs “MLLIB” in Apache Spark's documentation?
I am trying to figure out which pyspark library to use with Word2Vec and I'm presented with two options according to the pyspark documentation.
https://spark.apache.org/docs/2.2.0/mllib-feature-...
1
vote
0answers
190 views
cosine similarity between items (purchase data) and normalisation
I'm using IndexedRowMatrix which represents the products's user purchase behaviours and in order to build product recommendations, I use cosine similarity to calculate similarities between products. ...
0
votes
1answer
320 views
Accumulators in Spark (PySpark) without global variables?
BACKGROUND
Consider the following textbook example which uses accumulators to add vectors.
...
0
votes
1answer
612 views
PySpark: java.io.EOFException
System:
1 name node, 4 cores, 16 GB RAM
1 master node, 4 cores, 16 GB RAM
6 data nodes, 4 cores, 16 GB RAM each
6 worker nodes, 4 cores, 16 GB RAM each
around 5 Terabytes of storage space
The data ...
0
votes
1answer
904 views
Reliable way to verify Pyspark data frame column type
If I read data from a CSV, all the columns will be of "String" type by default. Generally, I inspect the data using the following functions which gives an overview of the data and its types
...
0
votes
1answer
2k views
Replacing null with average in pyspark
I have a problem during upsampling operation in PySpark. My dataframe is:
...
0
votes
1answer
1k views
Plotting in PySpark?
I have been searching for methods to plot in PySpark.
I couldn't find any resource on plotting data residing in DataFrame in PySpark. The only methods which are listed are:
through method ...
0
votes
2answers
63 views
Apache Spark Understanding
I have read a lot about Apache Spark. About RDD, DataFrames etc etc. But I haven't come across a simple explanation as to why and where should we be using Apache Spark? If my primary goal is machine/...
1
vote
0answers
316 views
Frequent itemset vs. frequent sequence mining algorithms on variables with an inherent order
I've been using Pyspark's PrefixSpan algorithm to mine sequences of variables with an inherent order (bus stops). I know that PrefixSpan takes into consideration the order of the items it is fed, ...
4
votes
2answers
216 views
k-means|| in PySpark
I'm trying to apply k-means$\|$ clustering in PySpark.
According to this paper, there is an oversampling factor, $l$, that would affect the model's cost.
I couldn't find any parameter regarding ...
1
vote
0answers
4k views
Adding multiple columns to spark dataframe
I would like to add several columns to a spark (actually pyspark) dataframe , these columns all being functions of several input columns in the df. There seems to be no 'add_columns' in spark, and ...
1
vote
0answers
107 views
Running H2O in databricks
I'm trying to run H2O in databricks. However, when I do the following:
hc = pysparkling.H2OContext.getOrCreate(spark)
I get the following error:
java.lang.AbstractMethodError
Does anyone know what ...
0
votes
2answers
269 views
Exponential runtime on distance matrix
I have an exploratory script running a Databricks notebook that performs a simple arithmetic function (Pythagorean theorem) on all possible pairwise combinations of a list of pairs of floats (akin to ...
3
votes
1answer
2k views
2
votes
1answer
33 views
PySpark Filter shows only 1 row
I have a 12 GB .csv file and I am trying to filter rows based on some value in a column under "AppName"
Doing this in Pyspark:
...
4
votes
2answers
840 views
What are the alternatives to Python + Spark (pyspark)?
I like Python, and I like Spark, but they don't fit very well together. In particular,
it is very hard to use python functions in spark (have to create JVM binding for function in python)
it is ...
1
vote
1answer
303 views
Use of Random Forest algorithm in PySpark for imputation
I am wondering how to use Random Forest algorithm for imputing missing values in a dataset. It is supposed to work well with missing values but I am not sure how those missing values are dealt with ...
2
votes
1answer
2k views
Run python script using spark-submit on windows 7
I'm trying to submit a python script using spark-submit on windows 7. I'm using spark 2.3.0 built for hadoop 2.7 . The python script is supposed to read data from a csv file and plot graphs. is it ...
1
vote
0answers
19 views
Recommender system: Give a feature more significance than another
I am trying to build a recommender system that predicts hotel prices based on a great number of features. I have a column representing the hotel rating out of 5 and another column indicating the ...
6
votes
1answer
6k views
PySpark dataframe repartition
What happens when we do repartition on a PySpark dataframe based on the column. For example
dataframe.repartition('id')
Does this moves the data with the similar ...
0
votes
2answers
88 views
KMeans for crime patterns
I'm trying to find the patterns in the crime records I have in a database.
I thought clustering would be a way to do it.
This is my (cooked up) dataset:
...
0
votes
2answers
1k views
Filling missing values with pyspark using a probability distribution
I want to fill missing values in my dataframe.
...
2
votes
2answers
11k views
Is there any way to read Xlsx file in pyspark?Also want to read strings of column from each columnName
pd is a panda module is one way of reading excel but its not available in my cluster. I want to read excel without pd module. Code1 and Code2 are two implementations i want in pyspark.
Code 1: ...
0
votes
1answer
78 views
Install Spark and Hadoop in the same machine [closed]
I'm trying to install Spark Apache and Hadoop in the same machine. Spark will be used to process data, and HDFS from Hadoop will be used to store data. I first started by installing spark and it ...
4
votes
1answer
1k views
Spark development on local machine with PyCharm
I am wondering what is the best practice other devs are using for their python spark jobs.
I am building a dev environment in which I am looking to write code in PyCharm with SparkContext pointing to ...
1
vote
0answers
136 views
Creating a user-based recommendation engine?
I have a data set of some questions and answers that users have completed by choices. I'm trying to build a recommendation engine to find similar users based on their answers on top of this.
So for ...
1
vote
0answers
451 views
How to do Feature Extraction using Apache Spark
I am a newbie to Machine Learning and I have to do feature extraction for a banking application to detect / predict fraudulent transactions.
I have come across few articles on Feature Extraction ...
-1
votes
1answer
931 views
Words from LDA output pyspark ml
I have built a pipelined lda model using pyspark's ml. The code is below:
tokenizer = Tokenizer(inputCol='formatted_transcript',
outputCol='words')
stopWordsRemover = StopWordsRemover(...
1
vote
1answer
4k views
Plot RDD data using a pyspark dataframe from csv file [closed]
I have just started working with pyspark on very large csv file. I am using Spark version 2.1.0. I want to read data from a .csv file and load it into a spark dataframe and then after filtering ...
1
vote
1answer
358 views
Pyspark code is not performant enough when compared to pure python alternative
I transformed the existing code which was in python pasted below was in pyspark.
Python code:
...
0
votes
2answers
50 views
Predicting the variable's score using predictive analytics
I am new to predictive analytics but I am good at programming (like Spark, R).
I have 10 variables and I know their score's for over a long period of time. I want to know which variable will have ...
1
vote
0answers
79 views
Predicting sales in unobserved locations
I am building a model that is trying to predict the 5 year sales production of a salesperson in every zip code in the United States. I have only 7 years of data. This task is one piece of a larger ...
3
votes
2answers
3k views
Is there any use to running Pandas on Spark?
I just started working with PySpark this week, and the instance I have access two has Pandas installed. But what use is having Pandas on Spark?
Isn't the whole purpose of running scripts on PySpark ...
1
vote
1answer
7k views
Dataframe request with groupBy
I'm a beginner in Spark and I want to calculate the average of number per name.
I have a JSON file with this information
...
4
votes
0answers
129 views
RDD of gziped files to “uncompressed” Dataframe [closed]
I have a bunch of compressed text files each line of which containing a JSON object. Simplified my workflow looks like this:
...
2
votes
2answers
3k views
spark item similarity recommendation
I would like to build an recommendation engine using spark's Mlib itemsimilarity as mentioned here LINK
But it seems spark do not have this algorithm any more and ...
