PySpark for Big Data and RAM usage

Question

I'm trying to figure out the best and most efficient method of handing ETL operations for big data. My question is this.

Say I have a table that is ~50 GB in size. In order to effectively transfer the data from this table from one source to another, specifically using PySpark, do I need to have more than 50 GB of RAM?

Thanks for your help.

Answer 1

All depends of partitioning of the input table.

Here is 2 approaches:

• So if u have only one single partition then u will have a single task/job that will use single core from your cluster and that will ultimately require more than 50GB RAM, otherwise you’ll run OOM. In case u have read the data as multi partitioned table then that 50GB will be sufficient because will require memory for the task/job for each partition to process.
• On top of all u can use batch processing for ETL and this is how is used in production because you do not allocate Petabytes of resources-memory to just process/transform/model petabytes of datasets/tables.

So short:

• For ETL-data prep: read data is done in parallel and by partitions and each partition should fit into executors memory (didn’t saw partition of 50Gb or Petabytes of data so far), so ETL is easy to do in batch and leveraging power of partitions, performing any transformation on any size of the dataset or table.

• For Modeling/ML/DS: When we look from modeling/ML/DS perspective all depends of the model applied, this is why not all models scale to spark and most successful models on spark are ensemble or stacked models leveraging sample/subsampling for modeling.

Note: if u are not satisfied with default configuration for partition size u can repartitioning spark dataframe by col/cols and same does apply for processing/transforming data (using cols or partitions).

Some resources: show partitions for tables AND approach to calculate size of dataset in spark