Questions tagged [pyspark]
The Spark Python API (PySpark) exposes the apache-spark programming model to Python.
102
questions
2
votes
1answer
35 views
How to split a CSV into a dataframe without newline
Currently I'm using pyspark to make my df from a csv. However when I take the data in, it puts each element on a new line. Is there any way to keep the elements separate, and keep them on the same ...
0
votes
0answers
21 views
Machine learning after Cosine Similarity for fuzzy name matching
I am following ING Bank's Large Scale Fuzzy Name Matching and I have implemented their awesome_cossim_topn to produce a dataframe of names and their associated similarities. I then added the other ...
1
vote
0answers
21 views
Feature importance using logistic regression in pyspark
I am using logistic regression in PySpark. I have after splitting train and test dataset
...
0
votes
0answers
15 views
Online learning/ Incremental modelling in pyspark
Can online learning of a model is possible in Pyspark? Does Mlib library in pyspark supports that. If so how it can be done
0
votes
1answer
24 views
Incremental modelling of kmeans in pyspark
I have a large dataset and trained the model with kmeans for the first time. I saved the model and pipeline used . Now again I started collecting data. After sufficient data is collected using old ...
2
votes
1answer
30 views
Calculating Rank Ordering Error Metric for implicit recommendation
I'm reading Collaborative Filtering for Implicit Feedback Datasets. On page 6 they detail their evaluation strategy, which they define as mean Expected Percentile Ranking with the following formula:
$...
0
votes
1answer
164 views
How to select multiple columns in a RDD with Spark (pySpark)?
Lets say I have a RDD that has comma delimited data. Each comma delimited value represents the amount of hours slept in the day of a week.
So for i.e. [8,7,6,7,8,8,5]
How can I manipulate the RDD so ...
0
votes
0answers
9 views
(Py)Spark infrastructure for use in Java/C# website
Lets say I have experience creating ML models in Python, and have decided on training my models on Spark using Pyspark. This will form part of an ML infrastructure for a website with a Java or C# ...
0
votes
0answers
38 views
Running LSTM model on a big data sample using pyspark
I was wondering how does one run an LSTM model on a big dataframe in pyspark. Ideally, one wants to run the model parallelly on different nodes of a spark cluster. But how does one do that?
0
votes
0answers
20 views
Unseing label in PySpark MultilayerPerceptronClassifier
I'm trying to perform classification with a MLP in PySpark:
...
0
votes
2answers
27 views
customer segmentation with unbalanced data
I am trying to do a customer segmentation on my transactional data and I am struggling a little bit on the best approach. Since it is an unsupervised model I can throw it to any algorithm and get some ...
0
votes
1answer
14 views
Which ML method for multiclass (non-binary) text classification should I choose (from SparkML)?
I am working on a quite big dataset that will be processed on the cluster, so this is why I am using PySpark for that purpose.
The presentable records of this dataset have a such structure:
...
0
votes
0answers
20 views
How to store subset of columns from a csv file?
I need to create a table in hive (or Impala) by reading from a csv file (named file.csv), ...
0
votes
0answers
29 views
Jupyter notebook pyspark error in %manage_spark
The environment is Jupyterhub on Centos using Pyspark kernel. When trying to run the below commands in JupyterLab I get errors. But the same commands work fine if I revert to regular Jupyter notebook ...
0
votes
0answers
18 views
Setting up Zeppelin to work with a Spark Cluster
I have made a spark cluster containing 2 workers and 1 master. I followed the following link to set up the spark clusters. After successfully setting up the spark clusters I wanted to connect it to ...
1
vote
1answer
30 views
0
votes
0answers
32 views
How to read binary video format in HDFS using pyspark?
We want to work on detection of pedestrians in a video using ML algorithm.But the video is in the HDFS file system.How can we read the binary video file format from HDFS using spark?
Please help me ...
0
votes
0answers
37 views
How to score AFT survival model in pyspark?
I am using AFTSurvivalRegression package in pyspark. I have a label which is time and censor which is 1 or 0 for those event that already happened or not as well as some features. I plugged them in ...
0
votes
0answers
16 views
how to run pyspark on Hadoop cluster, cache data, then read file using pandas on local machine via its URI?
I connect to a Hadoop cluster via vpn. I want to execute a script that makes sql call and saves the data to a file on the remote machine, then using its URI I want to read on my local running Jupiter ...
0
votes
1answer
30 views
Regarding join in pyspark
Given a rdd in pyspark, $A=[(0, 1), (1, 2), (1, 3), (3, 0)]$
I am trying to find self join of A such that the second element of previous pair is equal to first element of next pair i.e.
$...
0
votes
0answers
49 views
sorting pyspark dataframe
I'm sorting a pyspark DataFrame in databricks by df.sort(desc('x')). However, when I look at the DataFrame by display(df), I don't see the DataFrame being sorted in 'x'. What could be the reason?
0
votes
0answers
160 views
Python data cleaning for CSV using Pandas and PySpark and mapping to JSON in AWS ETL Glue Environment
I have recently started working on some ETL work and wanted some guidance in this area related to data cleaning from CSV to JSON mapping using AWS Glue, Python (pandas, pyspark).
Question
Need a ...
1
vote
1answer
521 views
PicklingError: Could not serialize object: TypeError: can't pickle fasttext_pybind.fasttext objects
I built a fasttext classification model in order to do sentiment analysis for facebook comments (using pyspark 2.4.1 on windows). When I use the prediction model function to predict the class of a ...
3
votes
0answers
133 views
Pyspark: Filter dataframe based on separate specific conditions
How can I select only certain entries that match my condition and from those entries, filter again using regex?
For instance, I have this dataframe (df)
...
1
vote
0answers
34 views
Where is $SPARK_HOME on an EC2 instance?
I am working on an EC2 instance with anaconda and pyspark pre-installed. My $SPARK_HOME variable is not set. Any ideas on how to locate my spark home?
1
vote
2answers
95 views
Record Linkage problem
I am building Matching Alogoritm using ML.Project is to match Internal customer data with external customer data.Features are names,address,city,state and zip.
We create pairs between data sets and ...
0
votes
0answers
5 views
PCAModel and PipelineModel, how to get explainedVariance
in pyspark PCAModel contains explainedVariance() method , but once you use Pipeline and specify PCA as a "stage", you will get a PipelineModel as an output and this one does not contain ...
2
votes
1answer
165 views
Pyspark Pipeline Custom Transformer
I'm having some trouble understanding the creation of custom transformers for Pyspark pipelines.
I am writing a custom transformer that will take the dataframe column Company and remove stray commas:
...
0
votes
0answers
16 views
Data sharing between Custom Tranformer and estimator in pyspark
Using Titanic Dataset for classification. Created custom Estimator and transformer for pipeline. some of logical operation need to be done in fit() method and data ...
0
votes
0answers
53 views
I got a overall mean average precision score of 0 for a recommendation engine
I just wanted to know if receiving an overall MAP score of 0 in a recommendation engine was possible, or a sign that my calculation or my logic for the engine was wrong.
0
votes
0answers
40 views
Code completion and suggestion EMR Notebook
I Created an EMR Cluster with Pyspark and connected to its notebook.
However i noticed there's no code completion neither method documentation.
Has anyone ever had the same issue? Didi you solve it?
...
4
votes
2answers
413 views
Model ensemble with Spark or Scikit Learn
I am using Spark MLLib to make prediction and I would like to know if it is possible to create your custom Estimators.
Here is a reproducible of what I would like my model to do with the Spark api
<...
0
votes
1answer
269 views
Getting parameters of the best model with crossvalidation in with SparkMLLib
I am having trouble accessing the parameters of estimators of model in SparkMLlib. More precisely my problem is:
I have a logistic regression model for which I want to find the best regularization ...
1
vote
2answers
323 views
K-modes implementation in pyspark
I'm looking for an implementation of k-modes in pyspark. I found this and this as implementations.
First, I tried implementing k-modes using the first link and faced issues. So I went ahead and tried ...
2
votes
0answers
48 views
Obtain learning curve of Gradient Boosted Tree model in PySpark
Currently there seems to be no method in PySpark of checkpointing the performance of a model at each gradient update. Is there a way to get the performance of a model at each gradient update so that a ...
0
votes
1answer
83 views
Null predictions for ALS in Pyspark
I am trying to read from my dataset which has three coloumns. (User, Repository and Number of Stars)
In[10]
...
0
votes
0answers
57 views
Can we use gunicorn to deploy pyspark models?
I have created a pyspark model and want to deploy it as a real time application. I am using Flask to handle requests and gunicorn to handle multiple requests by increasing workers and threads. To run ...
2
votes
1answer
2k views
1
vote
2answers
574 views
PySpark v Pandas Dataframe Memory Issue
Suppose I have a csv file with 20k rows, which I import into Pandas dataframe. I then run models like Random Forest or Logistic Regression from sklearn package and it runs fine. However, when I import ...
1
vote
0answers
199 views
What is the difference between PySpark's featuresCol, labelCol, predictionCol, and probabilityCol?
I am attempting to train a random forest classifier (pyspark.ml.classification.RandomForestClassifier) on a large dataset (~70gb). However, I am not sure what to ...
0
votes
1answer
383 views
Using keras tokenizer on Spark DataFrame. Column is not iterable [closed]
I am having trouble using the Tokenizer
The code is
...
1
vote
0answers
164 views
alternatives to pyspark to query a csv file locally (no distributed computation needed)
I am reading a csv file with pyspark to extract some information from it. I am running pyspark locally and I do not need distributed computation. On the contrary I need some plotting library such as ...
1
vote
0answers
14 views
Is there a built-in pyspark function for the paradigm union(filter, take)? [closed]
Essentially, filtering by key, taking a specific number (based on key) from the DataFrame, and unioning these results for a number of values.
For example,
...
1
vote
1answer
52 views
Best Data Science & Machine Learning Conferences 2019? [closed]
What are the best data science conferences you have attended? I am looking for a conference that will help me further develop my Data Science and Machine Learning competencies. Topics that are ...
0
votes
2answers
985 views
Why is there a difference of “ML” vs “MLLIB” in Apache Spark's documentation?
I am trying to figure out which pyspark library to use with Word2Vec and I'm presented with two options according to the pyspark documentation.
https://spark.apache.org/docs/2.2.0/mllib-feature-...
2
votes
1answer
453 views
cosine similarity between items (purchase data) and normalisation
I'm using IndexedRowMatrix which represents the products's user purchase behaviours and in order to build product recommendations, I use cosine similarity to calculate similarities between products. ...
0
votes
1answer
1k views
Accumulators in Spark (PySpark) without global variables?
BACKGROUND
Consider the following textbook example which uses accumulators to add vectors.
...
0
votes
1answer
2k views
PySpark: java.io.EOFException
System:
1 name node, 4 cores, 16 GB RAM
1 master node, 4 cores, 16 GB RAM
6 data nodes, 4 cores, 16 GB RAM each
6 worker nodes, 4 cores, 16 GB RAM each
around 5 Terabytes of storage space
The data ...
1
vote
3answers
4k views
Reliable way to verify Pyspark data frame column type
If I read data from a CSV, all the columns will be of "String" type by default. Generally, I inspect the data using the following functions which gives an overview of the data and its types
...
0
votes
1answer
4k views
Replacing null with average in pyspark
I have a problem during upsampling operation in PySpark. My dataframe is:
...
