Skip to main content

Questions tagged [pyspark]

The Spark Python API (PySpark) exposes the apache-spark programming model to Python.

Filter by
Sorted by
Tagged with
0 votes
0 answers
14 views

pyspark combined standard deviation?

I have a data frame which consists of multiple standard deviations with mean and count of each. my df is as follows ...
userDuR's user avatar
  • 101
0 votes
1 answer
24 views

Any Interface/Library that can take the Python ML code and run on spark cluster without learning PySpark?

I have been working with Python for machine learning and have a fair amount of code written in Python using libraries such as scikit-learn, pandas, and numpy. Recently, I’ve been faced with larger ...
Mohith7548's user avatar
0 votes
1 answer
232 views

convert a date string to utc timezone in pyspark

My input is "20220212" and I should get output like "2022-02-12T00:00:00+00:00" I have written the following ...
Monisha Sivanathan's user avatar
0 votes
0 answers
70 views

PySpark Logistic regression model weights are inconsistent between runs

I am training a pyspark logistic regression model using pyspark mllib. I am noticing that the weights are not being consistent in between runs. I have set the random seed in the training script and ...
hypothesisusable's user avatar
0 votes
0 answers
52 views

PySpark KMeans model has negative training cost

I am fitting a KMeans model for clustering using cosine distance measure. After model fitting, when I check the training cost of the model as below, ...
hypothesisusable's user avatar
0 votes
0 answers
28 views

Read from database every time a new record goes through a Spark pipeline

I'm using Spark to filter and transform every new record that goes through a data stream, the problem is that for each new record I need to read a table from a Cassandra database in order to have the ...
José Luis Benitez's user avatar
0 votes
0 answers
815 views

Using pyspark to create a large precomputed cosine similarity matrix from text data

I would like to precompute a cosine similarity matrix for a large dataset (upwards of 5 million rows) using pyspark. Here's what I have so far. libraries: ...
cfrench's user avatar
1 vote
1 answer
127 views

Is it valid to use Spark's StandardScaler on sparse input?

While I know it's possible to use StandardScaler on a SparseVector column, I wonder now if this is a valid transformation. My reason is that the output (most likely) will not be sparse. For example, ...
user12138762's user avatar
0 votes
0 answers
108 views

identify data type of column from table using NER models

I have structure data with csv or parquet format, I would like to extract the data type of the column by analyzing the data. when I looked at the NER from Hugging phase transformers, it actually ...
Anantha's user avatar
1 vote
0 answers
44 views

Not able to read data from Mongodb for below schema [closed]

Am trying to read very complex json from mongoDB. Tried in multiple ways nut no luck. Sample schema below : ...
sai's user avatar
  • 11
1 vote
0 answers
108 views

Why is GPU accelerated node much slower than CPU node for training a random forest model on databricks?

I have a dataset about 5 million rows with 14 features and a binary target. I decided to train a pyspark random forest classifier on Databricks. The CPU cluster I created contains 2 c4.8xlarge workers ...
Zhenyu Zhang's user avatar
2 votes
0 answers
1k views

Proper way to store a dict as a column value in parquet format using pyspark

I have a requirement to store a nested list of json objects in a column by doing a JOIN between two datasets related by one-to-many relation. Example: stackoverflow posts (each question can have one ...
vangap's user avatar
  • 121
1 vote
0 answers
56 views

Find closest item from ALS model using KNN

I have a dataset like: ...
Chris_007's user avatar
  • 193
0 votes
2 answers
2k views

Exploratory data analysis (EDA) on large dataset

I am working with lots of data (we have a table that produces 30 million rows daily). What is the best way to explore it (do on EDA)? Take a frictional slicing of the data randomly (100000 rows) or ...
MAS's user avatar
  • 35
0 votes
1 answer
641 views

Does one-hot encode effects chi-square test?

I am doing a feature selection for a data science project with one of those feature being a high cardinality categorical variable (for context, it’s nationality). I know chi-square test could handle ...
Reynard Ryanda's user avatar
0 votes
1 answer
12 views

Contunious model update

Hello I have a trained model on credit card fraud detection, however I want to find a solution so I can update the model parameters with new input records to simulate the pattern changes, is there any ...
hannibal Bra's user avatar
0 votes
1 answer
270 views

Group a spark dataframe by a starting event to an ending event

Given a series of events (with datetime) such as: failed, failed, passed, failed, passed, passed I want to retrieve the time from when it first "failed" to when it first "passed," ...
Ronen's user avatar
  • 101
0 votes
0 answers
205 views

Generalized Additive Modeling Apache Spark implementation

Does Spark MLlib support Generalized Additive Modeling? How does one go about implementing GAM models in Spark? I want to implement GAM (Generalized additive model) model in Spark. Based on my ...
Pavan's user avatar
  • 1
0 votes
1 answer
91 views

CREATE TABLE USING Oracle DATA_SOURCE

I am trying to create a table using ORACLE as a data source using spark query but getting an error. %sql CREATE TABLE TEST USING org.apache.spark.sql.jdbc OPTIONS ( url "jdbc:oracle:thin:@...
nikhil parmar's user avatar
0 votes
0 answers
57 views

Pyspark vs Pandas: When should we consider to change from Pandas to Pyspark

We trained a model in a single server using pandas, dataframe = 2.000.000 rows(I run it later in my own laptop), now we are migrating the code to the cloud and in that case to Databricks. In ...
Enrique Benito Casado's user avatar
0 votes
0 answers
275 views

Creating table in Databricks using the table from Oracle

I am trying to create a table in Databricks using a object in Oracle Db, but getting an error ...
nikhil parmar's user avatar
0 votes
1 answer
19 views

Best approach to find top-k most likely items

I am working with large datasets of papers and authors. I am trying to find top-k authors that are likely to cite a new paper on the unseen dataset (https://www.aminer.org/aminernetwork). My setup is ...
Lazar Gugleta's user avatar
1 vote
1 answer
286 views

How to get dummy variables from "first name"

I intend to predict the age of customers using some features. There are some categorical features that I need to convert to dummy variables before the modelling stage. Since the datasets are so big (...
Raha Moosavi's user avatar
1 vote
1 answer
368 views

PySpark for Big Data and RAM usage

I'm trying to figure out the best and most efficient method of handing ETL operations for big data. My question is this. Say I have a table that is ~50 GB in size. In order to effectively transfer the ...
Shaun's user avatar
  • 11
2 votes
1 answer
139 views

Where and how to do large scale supervised machine learning?

I'm beginner in ML and I have a large dataset that has 15 features with 6M rows, so it becomes challenging to work on it locally. I can train one model locally but to perform hyper parameter tuning ...
ro23's user avatar
  • 35
0 votes
1 answer
115 views

ML modeling a data with big amount of rows

I want to do ML modeling such XGboost, KNN, and similar models on data with 9 numerical features and more than 25 million rows and the size of data is almost 2.5 Gig and I prefer to use all the data ...
Amin Zadenoori's user avatar
1 vote
0 answers
39 views

Why do I need to create a dataframe just to add a column? [closed]

So I worked with a bit of pandas and in that, it's easy to add a column to an existing dataframe. But in spark, a new dataframe needs to be created using 'withColumn'. I'm a complete newbie so is ...
MashedPotoatoes's user avatar
1 vote
1 answer
161 views

Pyspark.read from MariaDB fetches duplicate header rows instead of data

It's a very strange issue- instead of fetching data like expected, it returns every row as a duplicate of the header. Every cell is a literal string of the column name. If I don't use the query like ...
Peace Blaster's user avatar
0 votes
1 answer
1k views

PySpark Crossvalidation error

I want to do a very simple cross validation using LogisticRegression. Here is my code: ...
Robse's user avatar
  • 103
2 votes
1 answer
255 views

Spark: How to run PCA parallelized? Only one thread used

I use pySpark and set my configuration like following: ...
bonfab's user avatar
  • 261
0 votes
0 answers
2k views

multi-label prediction with pySpark

I am new to Spark I am using pyspark to predict a multi label results. I have converted multi labels to binary So my labels will look like this ...
asmgx's user avatar
  • 549
1 vote
3 answers
10k views

PicklingError in pyspark (PicklingError: Can't pickle <class '__main__.Person'>: attribute lookup Person on __main__ failed)

I am unable to pickle the below class. I am using data bricks 6.5 ML (includes Apache Spark 2.4.5, Scala 2.11) ...
kamal kumawat's user avatar
0 votes
1 answer
74 views

Creating more than one worker nodes for local windows machine [closed]

I am using windows laptop. And I installed apache spark for my laptop. And I try to measure spark performance by changing spark components. because of that I want to create more than one worker nodes ...
randunu galhena's user avatar
4 votes
3 answers
12k views

Remove all columns where the entire column is null

I have a very dirty csv where there are several columns with only null values. I would like to remove them. I am trying to select all columns where the count of null values in the column is not equal ...
CJL's user avatar
  • 143
0 votes
1 answer
436 views

How to run Spark python code in Jupyter Notebook via command prompt

I am trying to import a data frame into spark using Python's pyspark module. For this, I used Jupyter Notebook and executed the code shown in the screenshot below After that I want to run this in CMD ...
randunu galhena's user avatar
1 vote
0 answers
1k views

cannot access hive from spark [closed]

I am trying to install a hadoop + spark + hive cluster. I am using hadoop 3.1.2, spark 2.4.5 (scala 2.11 prebuilt with user-provided hadoop) and hive 2.3.3 (also tried 3.1.2 with the exact same ...
user3044083's user avatar
0 votes
1 answer
710 views

Spark/Databricks: GPU does not appear to be utilized for ML regression (cross-validation, prediction) notebook

I have created and attached a notebook to a GPU-enabled Databricks cluster (6.4 ML (includes Apache Spark 2.4.5, GPU, Scala 2.11), EC2 type: p2.xlarge). I have started running the notebook that ...
James Adams's user avatar
1 vote
1 answer
274 views

Pyspark Dataframes to Pandas and ML Ops - Parallel Execution Hold?

If I convert a spark dataframe into a pandas dataframe and subsequently apply pandas operations and sklearn models to the dataset in databricks, will the operations from pandas and sklearn be ...
Soorya Paturi's user avatar
2 votes
3 answers
2k views

Is there a cost associated with converting Koalas dataframe to Spark dataframe?

I know that pandas works "under the hood" with numpy arrays stored in dictionaries. In contrast, Koalas works with the underlying Spark framework. Does that mean that there is no extra cost associated ...
DataBach's user avatar
  • 165
1 vote
2 answers
1k views

PySpark: How do I specify dropna axis in PySpark transformation?

I would like to drop columns that contain all null values using dropna(). With Pandas you can do this with setting the keyword argument ...
DataBach's user avatar
  • 165
0 votes
1 answer
781 views

How to apply K-Medoids in PySpark?

the pyspark ml library does not provide any clustering methods for K-Medoids. So my question is, how can one apply K-Medoids in a pyspark context?
EchoCache's user avatar
  • 111
1 vote
0 answers
20 views

Tool/Technology/Framework to process file with 1GB [closed]

I'm new to datascience, but I have the following problem: 1) Read a file of 1GB, which each line is a json object 2) There are two more files, much smaller, which I need to JOIN some data In this ...
Ramon Medeiros's user avatar
0 votes
1 answer
284 views

How to split a CSV into a dataframe without newline

Currently I'm using pyspark to make my df from a csv. However when I take the data in, it puts each element on a new line. Is there any way to keep the elements separate, and keep them on the same ...
Kelo's user avatar
  • 109
1 vote
0 answers
602 views

Feature importance using logistic regression in pyspark

I am using logistic regression in PySpark. I have after splitting train and test dataset ...
epsilones's user avatar
  • 121
0 votes
1 answer
312 views

Incremental modelling of kmeans in pyspark

I have a large dataset and trained the model with kmeans for the first time. I saved the model and pipeline used . Now again I started collecting data. After sufficient data is collected using old ...
Abhishek Diwate's user avatar
2 votes
1 answer
899 views

Calculating Rank Ordering Error Metric for implicit recommendation

I'm reading Collaborative Filtering for Implicit Feedback Datasets. On page 6 they detail their evaluation strategy, which they define as mean Expected Percentile Ranking with the following formula: $...
Dan Scally's user avatar
  • 1,754
4 votes
1 answer
16k views

How to select multiple columns in a RDD with Spark (pySpark)?

Lets say I have a RDD that has comma delimited data. Each comma delimited value represents the amount of hours slept in the day of a week. So for i.e. [8,7,6,7,8,8,5] How can I manipulate the RDD so ...
Kent Wong's user avatar
  • 143
0 votes
2 answers
219 views

customer segmentation with unbalanced data

I am trying to do a customer segmentation on my transactional data and I am struggling a little bit on the best approach. Since it is an unsupervised model I can throw it to any algorithm and get some ...
sf34200's user avatar
0 votes
1 answer
378 views

Which ML method for multiclass (non-binary) text classification should I choose (from SparkML)?

I am working on a quite big dataset that will be processed on the cluster, so this is why I am using PySpark for that purpose. The presentable records of this dataset have a such structure: ...
user83701's user avatar
1 vote
1 answer
161 views

How to store subset of columns from a csv file?

I need to create a table in hive (or Impala) by reading from a csv file (named file.csv), ...
Francesco Pegoraro's user avatar