Questions tagged [pyspark]

The Spark Python API (PySpark) exposes the apache-spark programming model to Python.

Filter by
Sorted by
Tagged with
1
vote
1answer
24 views

Pyspark Matrix Transformation

Let's assume I have the following dataframe in PySpark: ...
0
votes
0answers
26 views

How to read binary video format in HDFS using pyspark?

We want to work on detection of pedestrians in a video using ML algorithm.But the video is in the HDFS file system.How can we read the binary video file format from HDFS using spark? Please help me ...
0
votes
0answers
7 views

How to score AFT survival model in pyspark?

I am using AFTSurvivalRegression package in pyspark. I have a label which is time and censor which is 1 or 0 for those event that already happened or not as well as some features. I plugged them in ...
0
votes
0answers
13 views

how to run pyspark on Hadoop cluster, cache data, then read file using pandas on local machine via its URI?

I connect to a Hadoop cluster via vpn. I want to execute a script that makes sql call and saves the data to a file on the remote machine, then using its URI I want to read on my local running Jupiter ...
0
votes
1answer
22 views

Regarding join in pyspark

Given a rdd in pyspark, $A=[(0, 1), (1, 2), (1, 3), (3, 0)]$ I am trying to find self join of A such that the second element of previous pair is equal to first element of next pair i.e. $...
0
votes
0answers
21 views

sorting pyspark dataframe

I'm sorting a pyspark DataFrame in databricks by df.sort(desc('x')). However, when I look at the DataFrame by display(df), I don't see the DataFrame being sorted in 'x'. What could be the reason?
0
votes
0answers
60 views

Python data cleaning for CSV using Pandas and PySpark and mapping to JSON in AWS ETL Glue Environment

I have recently started working on some ETL work and wanted some guidance in this area related to data cleaning from CSV to JSON mapping using AWS Glue, Python (pandas, pyspark). Question Need a ...
0
votes
1answer
61 views

PicklingError: Could not serialize object: TypeError: can't pickle fasttext_pybind.fasttext objects

I built a fasttext classification model in order to do sentiment analysis for facebook comments (using pyspark 2.4.1 on windows). When I use the prediction model function to predict the class of a ...
3
votes
0answers
45 views

Pyspark: Filter dataframe based on separate specific conditions

How can I select only certain entries that match my condition and from those entries, filter again using regex? For instance, I have this dataframe (df) ...
1
vote
0answers
21 views

Where is $SPARK_HOME on an EC2 instance?

I am working on an EC2 instance with anaconda and pyspark pre-installed. My $SPARK_HOME variable is not set. Any ideas on how to locate my spark home?
1
vote
2answers
36 views

Record Linkage problem

I am building Matching Alogoritm using ML.Project is to match Internal customer data with external customer data.Features are names,address,city,state and zip. We create pairs between data sets and ...
0
votes
0answers
3 views

PCAModel and PipelineModel, how to get explainedVariance

in pyspark PCAModel contains explainedVariance() method , but once you use Pipeline and specify PCA as a "stage", you will get a PipelineModel as an output and this one does not contain ...
2
votes
0answers
24 views

Pyspark Pipeline Custom Transformer

I'm having some trouble understanding the creation of custom transformers for Pyspark pipelines. I am writing a custom transformer that will take the dataframe column Company and remove stray commas: ...
0
votes
0answers
14 views

Data sharing between Custom Tranformer and estimator in pyspark

Using Titanic Dataset for classification. Created custom Estimator and transformer for pipeline. some of logical operation need to be done in fit() method and data ...
0
votes
0answers
30 views

I got a overall mean average precision score of 0 for a recommendation engine

I just wanted to know if receiving an overall MAP score of 0 in a recommendation engine was possible, or a sign that my calculation or my logic for the engine was wrong.
0
votes
0answers
27 views

Code completion and suggestion EMR Notebook

I Created an EMR Cluster with Pyspark and connected to its notebook. However i noticed there's no code completion neither method documentation. Has anyone ever had the same issue? Didi you solve it? ...
4
votes
2answers
245 views

Model ensemble with Spark or Scikit Learn

I am using Spark MLLib to make prediction and I would like to know if it is possible to create your custom Estimators. Here is a reproducible of what I would like my model to do with the Spark api <...
0
votes
0answers
68 views

Getting parameters of the best model with crossvalidation in with SparkMLLib

I am having trouble accessing the parameters of estimators of model in SparkMLlib. More precisely my problem is: I have a logistic regression model for which I want to find the best regularization ...
0
votes
1answer
172 views

K-modes implementation in pyspark

I'm looking for an implementation of k-modes in pyspark. I found this and this as implementations. First, I tried implementing k-modes using the first link and faced issues. So I went ahead and tried ...
2
votes
0answers
32 views

Obtain learning curve of Gradient Boosted Tree model in PySpark

Currently there seems to be no method in PySpark of checkpointing the performance of a model at each gradient update. Is there a way to get the performance of a model at each gradient update so that a ...
0
votes
1answer
48 views

Null predictions for ALS in Pyspark

I am trying to read from my dataset which has three coloumns. (User, Repository and Number of Stars) In[10] ...
0
votes
0answers
45 views

Can we use gunicorn to deploy pyspark models?

I have created a pyspark model and want to deploy it as a real time application. I am using Flask to handle requests and gunicorn to handle multiple requests by increasing workers and threads. To run ...
2
votes
1answer
524 views

Spark DataFrame Limit Function Takes Too Much Time to Show

...
1
vote
2answers
204 views

PySpark v Pandas Dataframe Memory Issue

Suppose I have a csv file with 20k rows, when I import in Pandas dataframe format and run the ML algos like Random Forest or Logistic Regression from sklearn package it just runs fine. But same thing ...
0
votes
0answers
115 views

What is the difference between PySpark's featuresCol, labelCol, predictionCol, and probabilityCol?

I am attempting to train a random forest classifier (pyspark.ml.classification.RandomForestClassifier) on a large dataset (~70gb). However, I am not sure what to ...
0
votes
1answer
277 views

Using keras tokenizer on Spark DataFrame. Column is not iterable [closed]

I am having trouble using the Tokenizer The code is ...
1
vote
0answers
160 views

alternatives to pyspark to query a csv file locally (no distributed computation needed)

I am reading a csv file with pyspark to extract some information from it. I am running pyspark locally and I do not need distributed computation. On the contrary I need some plotting library such as ...
1
vote
0answers
13 views

Is there a built-in pyspark function for the paradigm union(filter, take)? [closed]

Essentially, filtering by key, taking a specific number (based on key) from the DataFrame, and unioning these results for a number of values. For example, ...
1
vote
1answer
49 views

Best Data Science & Machine Learning Conferences 2019? [closed]

What are the best data science conferences you have attended? I am looking for a conference that will help me further develop my Data Science and Machine Learning competencies. Topics that are ...
0
votes
2answers
443 views

Why is there a difference of “ML” vs “MLLIB” in Apache Spark's documentation?

I am trying to figure out which pyspark library to use with Word2Vec and I'm presented with two options according to the pyspark documentation. https://spark.apache.org/docs/2.2.0/mllib-feature-...
2
votes
1answer
337 views

cosine similarity between items (purchase data) and normalisation

I'm using IndexedRowMatrix which represents the products's user purchase behaviours and in order to build product recommendations, I use cosine similarity to calculate similarities between products. ...
0
votes
1answer
643 views

Accumulators in Spark (PySpark) without global variables?

BACKGROUND Consider the following textbook example which uses accumulators to add vectors. ...
0
votes
1answer
2k views

PySpark: java.io.EOFException

System: 1 name node, 4 cores, 16 GB RAM 1 master node, 4 cores, 16 GB RAM 6 data nodes, 4 cores, 16 GB RAM each 6 worker nodes, 4 cores, 16 GB RAM each around 5 Terabytes of storage space The data ...
1
vote
3answers
2k views

Reliable way to verify Pyspark data frame column type

If I read data from a CSV, all the columns will be of "String" type by default. Generally, I inspect the data using the following functions which gives an overview of the data and its types ...
0
votes
1answer
3k views

Replacing null with average in pyspark

I have a problem during upsampling operation in PySpark. My dataframe is: ...
1
vote
1answer
4k views

Plotting in PySpark?

I have been searching for methods to plot in PySpark. I couldn't find any resource on plotting data residing in DataFrame in PySpark. The only methods which are listed are: through method ...
0
votes
2answers
85 views

Apache Spark Understanding

I have read a lot about Apache Spark. About RDD, DataFrames etc etc. But I haven't come across a simple explanation as to why and where should we be using Apache Spark? If my primary goal is machine/...
1
vote
0answers
468 views

Frequent itemset vs. frequent sequence mining algorithms on variables with an inherent order

I've been using Pyspark's PrefixSpan algorithm to mine sequences of variables with an inherent order (bus stops). I know that PrefixSpan takes into consideration the order of the items it is fed, ...
4
votes
2answers
352 views

k-means|| in PySpark

I'm trying to apply k-means$\|$ clustering in PySpark. According to this paper, there is an oversampling factor, $l$, that would affect the model's cost. I couldn't find any parameter regarding ...
1
vote
0answers
7k views

Adding multiple columns to spark dataframe

I would like to add several columns to a spark (actually pyspark) dataframe , these columns all being functions of several input columns in the df. There seems to be no 'add_columns' in spark, and ...
1
vote
0answers
130 views

Running H2O in databricks

I'm trying to run H2O in databricks. However, when I do the following: hc = pysparkling.H2OContext.getOrCreate(spark) I get the following error: java.lang.AbstractMethodError Does anyone know what ...
0
votes
2answers
464 views

Exponential runtime on distance matrix

I have an exploratory script running a Databricks notebook that performs a simple arithmetic function (Pythagorean theorem) on all possible pairwise combinations of a list of pairs of floats (akin to ...
3
votes
1answer
3k views

convert list of tuple of tuple to list of tuple in pySpark

Code: ...
2
votes
1answer
41 views

PySpark Filter shows only 1 row

I have a 12 GB .csv file and I am trying to filter rows based on some value in a column under "AppName" Doing this in Pyspark: ...
4
votes
2answers
1k views

What are the alternatives to Python + Spark (pyspark)?

I like Python, and I like Spark, but they don't fit very well together. In particular, it is very hard to use python functions in spark (have to create JVM binding for function in python) it is ...
1
vote
1answer
379 views

Use of Random Forest algorithm in PySpark for imputation

I am wondering how to use Random Forest algorithm for imputing missing values in a dataset. It is supposed to work well with missing values but I am not sure how those missing values are dealt with ...
2
votes
1answer
2k views

Run python script using spark-submit on windows 7

I'm trying to submit a python script using spark-submit on windows 7. I'm using spark 2.3.0 built for hadoop 2.7 . The python script is supposed to read data from a csv file and plot graphs. is it ...
1
vote
0answers
20 views

Recommender system: Give a feature more significance than another

I am trying to build a recommender system that predicts hotel prices based on a great number of features. I have a column representing the hotel rating out of 5 and another column indicating the ...
6
votes
1answer
9k views

PySpark dataframe repartition

What happens when we do repartition on a PySpark dataframe based on the column. For example dataframe.repartition('id') Does this moves the data with the similar ...
0
votes
2answers
104 views

KMeans for crime patterns

I'm trying to find the patterns in the crime records I have in a database. I thought clustering would be a way to do it. This is my (cooked up) dataset: ...