Questions tagged [pyspark]

The Spark Python API (PySpark) exposes the apache-spark programming model to Python.

Filter by
Sorted by
Tagged with
0
votes
0answers
18 views

How to implement predict_proba() with spark.ml logistic regression

In sklearn it is possible to use the predict_proba() function to obtain the probability matrix. Since in ...
0
votes
0answers
57 views

Access global variable from UDF (User Defined Function) in python in spark

I am trying to alter a global variable from inside a pyspark.sql.functions.udf function in python. But, the change in not getting reflected in the global variable. ...
1
vote
1answer
14 views

PySpark for Big Data and RAM usage

I'm trying to figure out the best and most efficient method of handing ETL operations for big data. My question is this. Say I have a table that is ~50 GB in size. In order to effectively transfer the ...
2
votes
1answer
126 views

Where and how to do large scale supervised machine learning?

I'm beginner in ML and I have a large dataset that has 15 features with 6M rows, so it becomes challenging to work on it locally. I can train one model locally but to perform hyper parameter tuning ...
1
vote
1answer
39 views

ML modeling a data with big amount of rows

I want to do ML modeling such XGboost, KNN, and similar models on data with 9 numerical features and more than 25 million rows and the size of data is almost 2.5 Gig and I prefer to use all the data ...
1
vote
0answers
28 views

Why do I need to create a dataframe just to add a column? [closed]

So I worked with a bit of pandas and in that, it's easy to add a column to an existing dataframe. But in spark, a new dataframe needs to be created using 'withColumn'. I'm a complete newbie so is ...
1
vote
1answer
32 views

Pyspark.read from MariaDB fetches duplicate header rows instead of data

It's a very strange issue- instead of fetching data like expected, it returns every row as a duplicate of the header. Every cell is a literal string of the column name. If I don't use the query like ...
0
votes
0answers
8 views

Run ML model on a distributed manner on dataframe rdd partitions

When running a logistic regression on a large dataset with repartition, how do we stitch the results of the each partition results back to a dataframe to analyze. ...
0
votes
0answers
12 views

time syncronisation

Introduction Given is a vehicle which is moving past a station. If the vehicle passes the station (for simplicity determined by GPS) it sends RF-signals, to the station saying "Hello I am here ...
0
votes
0answers
14 views

Network flow dataset labeling

I have a network flow dataset with 500 mio. records that looks similar to this (highly simplified) ...
0
votes
1answer
79 views

PySpark Crossvalidation error

I want to do a very simple cross validation using LogisticRegression. Here is my code: ...
1
vote
0answers
288 views

Flattening of a structure field in pyspark is slow?

I am rather new to (py)spark. I am using a python UDF to compute several outputs. My idea was to pack all the values in a StructType and then flatten the structure before collecting the result. The ...
2
votes
0answers
61 views

Spark: How to run PCA parallelized? Only one thread used

I use pySpark and set my configuration like following: ...
0
votes
0answers
642 views

multi-label prediction with pySpark

I am new to Spark I am using pyspark to predict a multi label results. I have converted multi labels to binary So my labels will look like this ...
1
vote
1answer
481 views

PicklingError in pyspark (PicklingError: Can't pickle <class '__main__.Person'>: attribute lookup Person on __main__ failed)

I am unable to pickle the below class. I am using data bricks 6.5 ML (includes Apache Spark 2.4.5, Scala 2.11) ...
0
votes
1answer
21 views

Creating more than one worker nodes for local windows machine [closed]

I am using windows laptop. And I installed apache spark for my laptop. And I try to measure spark performance by changing spark components. because of that I want to create more than one worker nodes ...
1
vote
3answers
3k views

Remove all columns where the entire column is null

I have a very dirty csv where there are several columns with only null values. I would like to remove them. I am trying to select all columns where the count of null values in the column is not equal ...
0
votes
1answer
50 views

How to run Spark python code in Jupyter Notebook via command prompt

I am trying to import a data frame into spark using Python's pyspark module. For this, I used Jupyter Notebook and executed the code shown in the screenshot below After that I want to run this in CMD ...
1
vote
0answers
891 views

cannot access hive from spark [closed]

I am trying to install a hadoop + spark + hive cluster. I am using hadoop 3.1.2, spark 2.4.5 (scala 2.11 prebuilt with user-provided hadoop) and hive 2.3.3 (also tried 3.1.2 with the exact same ...
0
votes
1answer
158 views

Spark/Databricks: GPU does not appear to be utilized for ML regression (cross-validation, prediction) notebook

I have created and attached a notebook to a GPU-enabled Databricks cluster (6.4 ML (includes Apache Spark 2.4.5, GPU, Scala 2.11), EC2 type: p2.xlarge). I have started running the notebook that ...
1
vote
1answer
136 views

Pyspark Dataframes to Pandas and ML Ops - Parallel Execution Hold?

If I convert a spark dataframe into a pandas dataframe and subsequently apply pandas operations and sklearn models to the dataset in databricks, will the operations from pandas and sklearn be ...
1
vote
3answers
1k views

Is there a cost associated with converting Koalas dataframe to Spark dataframe?

I know that pandas works "under the hood" with numpy arrays stored in dictionaries. In contrast, Koalas works with the underlying Spark framework. Does that mean that there is no extra cost associated ...
2
votes
2answers
748 views

PySpark: How do I specify dropna axis in PySpark transformation?

I would like to drop columns that contain all null values using dropna(). With Pandas you can do this with setting the keyword argument ...
0
votes
1answer
250 views

How to apply K-Medoids in PySpark?

the pyspark ml library does not provide any clustering methods for K-Medoids. So my question is, how can one apply K-Medoids in a pyspark context?
1
vote
0answers
19 views

Tool/Technology/Framework to process file with 1GB [closed]

I'm new to datascience, but I have the following problem: 1) Read a file of 1GB, which each line is a json object 2) There are two more files, much smaller, which I need to JOIN some data In this ...
0
votes
1answer
194 views

How to split a CSV into a dataframe without newline

Currently I'm using pyspark to make my df from a csv. However when I take the data in, it puts each element on a new line. Is there any way to keep the elements separate, and keep them on the same ...
1
vote
0answers
307 views

Feature importance using logistic regression in pyspark

I am using logistic regression in PySpark. I have after splitting train and test dataset ...
0
votes
1answer
79 views

Incremental modelling of kmeans in pyspark

I have a large dataset and trained the model with kmeans for the first time. I saved the model and pipeline used . Now again I started collecting data. After sufficient data is collected using old ...
2
votes
1answer
385 views

Calculating Rank Ordering Error Metric for implicit recommendation

I'm reading Collaborative Filtering for Implicit Feedback Datasets. On page 6 they detail their evaluation strategy, which they define as mean Expected Percentile Ranking with the following formula: $...
2
votes
1answer
7k views

How to select multiple columns in a RDD with Spark (pySpark)?

Lets say I have a RDD that has comma delimited data. Each comma delimited value represents the amount of hours slept in the day of a week. So for i.e. [8,7,6,7,8,8,5] How can I manipulate the RDD so ...
0
votes
2answers
103 views

customer segmentation with unbalanced data

I am trying to do a customer segmentation on my transactional data and I am struggling a little bit on the best approach. Since it is an unsupervised model I can throw it to any algorithm and get some ...
1
vote
1answer
132 views

Which ML method for multiclass (non-binary) text classification should I choose (from SparkML)?

I am working on a quite big dataset that will be processed on the cluster, so this is why I am using PySpark for that purpose. The presentable records of this dataset have a such structure: ...
0
votes
1answer
95 views

How to store subset of columns from a csv file?

I need to create a table in hive (or Impala) by reading from a csv file (named file.csv), ...
1
vote
1answer
195 views

Pyspark Matrix Transformation

Let's assume I have the following dataframe in PySpark: ...
1
vote
0answers
54 views

How to read binary video format in HDFS using pyspark?

We want to work on detection of pedestrians in a video using ML algorithm.But the video is in the HDFS file system.How can we read the binary video file format from HDFS using spark? Please help me ...
0
votes
1answer
53 views

Regarding join in pyspark

Given a rdd in pyspark, $A=[(0, 1), (1, 2), (1, 3), (3, 0)]$ I am trying to find self join of A such that the second element of previous pair is equal to first element of next pair i.e. $...
3
votes
1answer
4k views

PicklingError: Could not serialize object: TypeError: can't pickle fasttext_pybind.fasttext objects

I built a fasttext classification model in order to do sentiment analysis for facebook comments (using pyspark 2.4.1 on windows). When I use the prediction model function to predict the class of a ...
6
votes
1answer
353 views

Pyspark: Filter dataframe based on separate specific conditions

How can I select only certain entries that match my condition and from those entries, filter again using regex? For instance, I have this dataframe (df) ...
0
votes
0answers
72 views

Where is $SPARK_HOME on an EC2 instance?

I am working on an EC2 instance with anaconda and pyspark pre-installed. My $SPARK_HOME variable is not set. Any ideas on how to locate my spark home?
1
vote
2answers
223 views

Record Linkage problem

I am building Matching Alogoritm using ML.Project is to match Internal customer data with external customer data.Features are names,address,city,state and zip. We create pairs between data sets and ...
3
votes
1answer
621 views

Pyspark Pipeline Custom Transformer

I'm having some trouble understanding the creation of custom transformers for Pyspark pipelines. I am writing a custom transformer that will take the dataframe column Company and remove stray commas: ...
0
votes
0answers
128 views

I got a overall mean average precision score of 0 for a recommendation engine

I just wanted to know if receiving an overall MAP score of 0 in a recommendation engine was possible, or a sign that my calculation or my logic for the engine was wrong.
6
votes
2answers
962 views

Model ensemble with Spark or Scikit Learn

I am using Spark MLLib to make prediction and I would like to know if it is possible to create your custom Estimators. Here is a reproducible of what I would like my model to do with the Spark api <...
1
vote
1answer
1k views

Getting parameters of the best model with crossvalidation in with SparkMLLib

I am having trouble accessing the parameters of estimators of model in SparkMLlib. More precisely my problem is: I have a logistic regression model for which I want to find the best regularization ...
3
votes
2answers
1k views

K-modes implementation in pyspark

I'm looking for an implementation of k-modes in pyspark. I found this and this as implementations. First, I tried implementing k-modes using the first link and faced issues. So I went ahead and tried ...
2
votes
0answers
75 views

Obtain learning curve of Gradient Boosted Tree model in PySpark

Currently there seems to be no method in PySpark of checkpointing the performance of a model at each gradient update. Is there a way to get the performance of a model at each gradient update so that a ...
1
vote
1answer
239 views

Null predictions for ALS in Pyspark

I am trying to read from my dataset which has three coloumns. (User, Repository and Number of Stars) In[10] ...
3
votes
1answer
9k views

Spark DataFrame “Limit” function takes too much time to display result

...
1
vote
2answers
2k views

PySpark v Pandas Dataframe Memory Issue

Suppose I have a csv file with 20k rows, which I import into Pandas dataframe. I then run models like Random Forest or Logistic Regression from sklearn package and it runs fine. However, when I import ...
2
votes
0answers
575 views

What is the difference between PySpark's featuresCol, labelCol, predictionCol, and probabilityCol?

I am attempting to train a random forest classifier (pyspark.ml.classification.RandomForestClassifier) on a large dataset (~70gb). However, I am not sure what to ...