New answers tagged

1

All depends of partitioning of the input table. Here is 2 approaches: So if u have only one single partition then u will have a single task/job that will use single core from your cluster and that will ultimately require more than 50GB RAM, otherwise you’ll run OOM. In case u have read the data as multi partitioned table then that 50GB will be sufficient ...


5

First, when working with big data most of the time it's more convenient to work with a random subset rather than the whole thing: usually during the design and testing stages there is no need to work with the full data since optimal performance is not needed. Second, it's often useful to do an ablation study in order to check that using the full data is ...


0

Regarding cloud services for 2.5 Gig - all have instances up to 64G so you are fine to go with either of AWS / GCP or others - that is not your problem. What you need to do is to make sure you are loading your data into python efficiently. what are these 9 numerical features ? integers ? binary float ? how many digit of precision do you need ? If values are ...


Top 50 recent answers are included