Questions tagged [pyspark]
The Spark Python API (PySpark) exposes the apache-spark programming model to Python.
132
questions
0
votes
0
answers
216
views
How to dynamically add columns in Dataframe Spark by using a config.ini file parameters
I am learning to use Pyspark and at work I have been assigned a task that I can not solve for now.
There is a config.ini file where there are a series of ...
- 101
0
votes
0
answers
13
views
Extracting variables from different properties with similar timestamps -- PySpark
I have a list of variables I need to extract. I am using the PySpark API. Almost all variables are already in the same property, but just one variable I also need was logged in another property. For ...
- 101
1
vote
1
answer
28
views
Is it valid to use Spark's StandardScaler on sparse input?
While I know it's possible to use StandardScaler on a SparseVector column, I wonder now if this is a valid transformation. My reason is that the output (most likely) will not be sparse. For example, ...
- 13
0
votes
0
answers
31
views
identify data type of column from table using NER models
I have structure data with csv or parquet format, I would like to extract the data type of the column by analyzing the data.
when I looked at the NER from Hugging phase transformers, it actually ...
- 1
0
votes
0
answers
96
views
Dimensionality Reduction of Categorical Variables in Spark
First off, I hope this question hasn't been asked already. I've found questions regarding the use of PCA vs. MCA in the reduction of categorical variables, but I've yet to see a solution as it ...
- 1
1
vote
0
answers
16
views
Not able to read data from Mongodb for below schema [closed]
Am trying to read very complex json from mongoDB. Tried in multiple ways nut no luck. Sample schema below :
...
- 11
1
vote
0
answers
23
views
Why is GPU accelerated node much slower than CPU node for training a random forest model on databricks?
I have a dataset about 5 million rows with 14 features and a binary target. I decided to train a pyspark random forest classifier on Databricks. The CPU cluster I created contains 2 c4.8xlarge workers ...
- 13
0
votes
0
answers
134
views
Proper way to store a dict as a column value in parquet format using pyspark
I have a requirement to store a nested list of json objects in a column by doing a JOIN between two datasets related by one-to-many relation.
Example: stackoverflow posts (each question can have one ...
- 101
0
votes
0
answers
25
views
Pyspark install failing on Mac OSX
Not able to run pyspark, spark-shell, apache-spark on my mac after attempting install.
At ...
- 101
0
votes
0
answers
141
views
what is difference between greatest and max function in spark data frame?
What is the exact difference between greatest and max functions in Spark data frame ,Since it returns the max value in the specified range
0
votes
0
answers
185
views
Spark - Group by coulmn_1, and calculate the precentiles of coulmn_2 where the precentile to calculate is in coulmn_3
I have a data frame with 3 main columns.
coulmn_1 - the column I want to preform group by on
coulmn_2 - contains numeric values I would like to calculate percentiles on
coulmn_3 - contains a number ...
- 101
1
vote
0
answers
24
views
0
votes
0
answers
45
views
How do you iterate an entire query based on a condition calculated from the result of the query in Spark/pyspark?
I am writing a pyspark program to iterate an entire query many times. A mix of relational and functional spark transformations get applied to a dataframe and produce another dataframe with exactly the ...
- 1
0
votes
0
answers
43
views
How does ALS implementation calculate ratings when model.transform is called?
The spark ALS model is based on this paper: Collaborative Filtering for Implicit Feedback datasets. .
Here, latent vectors are learnt such that instead of estimating R (ratings matrix), they only ...
- 101
0
votes
2
answers
880
views
Exploratory data analysis (EDA) on large dataset
I am working with lots of data (we have a table that produces 30 million rows daily).
What is the best way to explore it (do on EDA)?
Take a frictional slicing of the data randomly (100000 rows)
or ...
- 35
0
votes
1
answer
306
views
Does one-hot encode effects chi-square test?
I am doing a feature selection for a data science project with one of those feature being a high cardinality categorical variable (for context, it’s nationality). I know chi-square test could handle ...
0
votes
1
answer
10
views
Contunious model update
Hello I have a trained model on credit card fraud detection, however I want to find a solution so I can update the model parameters with new input records to simulate the pattern changes, is there any ...
0
votes
0
answers
194
views
Computing number of business days between start/end columns
I have two Dataframes
facts:
columns: data, start_date and end_date
holidays:
column: <...
- 101
0
votes
0
answers
55
views
PySpark createDataFrame() throws segmentation fault on Mac
I'm trying to learn PySpark. Finally got it installed following the tutorial here: https://sparkbyexamples.com/pyspark/install-pyspark-in-anaconda-jupyter-notebook/
However, even though I am now able ...
0
votes
0
answers
313
views
Is it possible to implement an rdd version of a for loop having map and reduce using pyspark?
I need to test an algorithm that computes a function on a dataframe where in each execution I drop a column and computes the function. This is a example in python pyspark but without using rdd:
...
- 1
0
votes
1
answer
107
views
Group a spark dataframe by a starting event to an ending event
Given a series of events (with datetime) such as:
failed, failed, passed, failed, passed, passed
I want to retrieve the time from when it first "failed" to when it first "passed," ...
- 101
0
votes
0
answers
101
views
Generalized Additive Modeling Apache Spark implementation
Does Spark MLlib support Generalized Additive Modeling? How does one go about implementing GAM models in Spark?
I want to implement GAM (Generalized additive model) model in Spark. Based on my ...
- 1
0
votes
1
answer
71
views
CREATE TABLE USING Oracle DATA_SOURCE
I am trying to create a table using ORACLE as a data source using spark query but getting an error.
%sql
CREATE TABLE TEST
USING org.apache.spark.sql.jdbc
OPTIONS (
url "jdbc:oracle:thin:@...
0
votes
0
answers
45
views
Pyspark vs Pandas: When should we consider to change from Pandas to Pyspark
We trained a model in a single server using pandas, dataframe = 2.000.000 rows(I run it later in my own laptop), now we are migrating the code to the cloud and in that case to Databricks.
In ...
0
votes
0
answers
182
views
Creating table in Databricks using the table from Oracle
I am trying to create a table in Databricks using a object in Oracle Db, but getting an error
...
0
votes
1
answer
17
views
Best approach to find top-k most likely items
I am working with large datasets of papers and authors.
I am trying to find top-k authors that are likely to cite a new paper on the unseen dataset (https://www.aminer.org/aminernetwork).
My setup is ...
1
vote
1
answer
215
views
How to get dummy variables from "first name"
I intend to predict the age of customers using some features. There are some categorical features that I need to convert to dummy variables before the modelling stage.
Since the datasets are so big (...
- 111
1
vote
1
answer
198
views
PySpark for Big Data and RAM usage
I'm trying to figure out the best and most efficient method of handing ETL operations for big data. My question is this.
Say I have a table that is ~50 GB in size. In order to effectively transfer the ...
- 11
2
votes
1
answer
129
views
Where and how to do large scale supervised machine learning?
I'm beginner in ML and I have a large dataset that has 15 features with 6M rows, so it becomes challenging to work on it locally. I can train one model locally but to perform hyper parameter tuning ...
- 35
0
votes
1
answer
62
views
ML modeling a data with big amount of rows
I want to do ML modeling such XGboost, KNN, and similar models on data with 9 numerical features and more than 25 million rows and the size of data is almost 2.5 Gig and I prefer to use all the data ...
1
vote
0
answers
37
views
Why do I need to create a dataframe just to add a column? [closed]
So I worked with a bit of pandas and in that, it's easy to add a column to an existing dataframe. But in spark, a new dataframe needs to be created using 'withColumn'. I'm a complete newbie so is ...
1
vote
1
answer
111
views
Pyspark.read from MariaDB fetches duplicate header rows instead of data
It's a very strange issue- instead of fetching data like expected, it returns every row as a duplicate of the header. Every cell is a literal string of the column name.
If I don't use the query like ...
0
votes
1
answer
772
views
PySpark Crossvalidation error
I want to do a very simple cross validation using LogisticRegression.
Here is my code:
...
- 103
2
votes
1
answer
180
views
Spark: How to run PCA parallelized? Only one thread used
I use pySpark and set my configuration like following:
...
- 241
0
votes
0
answers
1k
views
multi-label prediction with pySpark
I am new to Spark
I am using pyspark to predict a multi label results.
I have converted multi labels to binary
So my labels will look like this
...
- 517
1
vote
3
answers
6k
views
PicklingError in pyspark (PicklingError: Can't pickle <class '__main__.Person'>: attribute lookup Person on __main__ failed)
I am unable to pickle the below class. I am using data bricks 6.5 ML (includes Apache Spark 2.4.5, Scala 2.11)
...
0
votes
1
answer
41
views
Creating more than one worker nodes for local windows machine [closed]
I am using windows laptop. And I installed apache spark for my laptop. And I try to measure spark performance by changing spark components. because of that I want to create more than one worker nodes ...
2
votes
3
answers
10k
views
Remove all columns where the entire column is null
I have a very dirty csv where there are several columns with only null values.
I would like to remove them. I am trying to select all columns where the count of null values in the column is not equal ...
- 123
0
votes
1
answer
291
views
How to run Spark python code in Jupyter Notebook via command prompt
I am trying to import a data frame into spark using Python's pyspark module. For this, I used Jupyter Notebook and executed the code shown in the screenshot below
After that I want to run this in CMD ...
1
vote
0
answers
1k
views
cannot access hive from spark [closed]
I am trying to install a hadoop + spark + hive cluster.
I am using hadoop 3.1.2, spark 2.4.5 (scala 2.11 prebuilt with user-provided hadoop) and hive 2.3.3 (also tried 3.1.2 with the exact same ...
- 11
0
votes
1
answer
397
views
Spark/Databricks: GPU does not appear to be utilized for ML regression (cross-validation, prediction) notebook
I have created and attached a notebook to a GPU-enabled Databricks cluster (6.4 ML (includes Apache Spark 2.4.5, GPU, Scala 2.11), EC2 type: p2.xlarge).
I have started running the notebook that ...
- 103
1
vote
1
answer
221
views
Pyspark Dataframes to Pandas and ML Ops - Parallel Execution Hold?
If I convert a spark dataframe into a pandas dataframe and subsequently apply pandas operations and sklearn models to the dataset in databricks, will the operations from pandas and sklearn be ...
2
votes
3
answers
1k
views
Is there a cost associated with converting Koalas dataframe to Spark dataframe?
I know that pandas works "under the hood" with numpy arrays stored in dictionaries. In contrast, Koalas works with the underlying Spark framework. Does that mean that there is no extra cost associated ...
- 165
1
vote
2
answers
1k
views
PySpark: How do I specify dropna axis in PySpark transformation?
I would like to drop columns that contain all null values using dropna(). With Pandas you can do this with setting the keyword argument ...
- 165
0
votes
1
answer
590
views
How to apply K-Medoids in PySpark?
the pyspark ml library does not provide any clustering methods for K-Medoids. So my question is, how can one apply K-Medoids in a pyspark context?
- 111
1
vote
0
answers
20
views
Tool/Technology/Framework to process file with 1GB [closed]
I'm new to datascience, but I have the following problem:
1) Read a file of 1GB, which each line is a json object
2) There are two more files, much smaller, which I need to JOIN some data
In this ...
- 113
0
votes
1
answer
238
views
How to split a CSV into a dataframe without newline
Currently I'm using pyspark to make my df from a csv. However when I take the data in, it puts each element on a new line. Is there any way to keep the elements separate, and keep them on the same ...
- 109
1
vote
0
answers
520
views
Feature importance using logistic regression in pyspark
I am using logistic regression in PySpark. I have after splitting train and test dataset
...
- 121
0
votes
1
answer
263
views
Incremental modelling of kmeans in pyspark
I have a large dataset and trained the model with kmeans for the first time. I saved the model and pipeline used . Now again I started collecting data. After sufficient data is collected using old ...
2
votes
1
answer
734
views
Calculating Rank Ordering Error Metric for implicit recommendation
I'm reading Collaborative Filtering for Implicit Feedback Datasets. On page 6 they detail their evaluation strategy, which they define as mean Expected Percentile Ranking with the following formula:
$...
- 1,714