Hot answers tagged

4

There are 2 things you can do here: 1.) Use libraries like Dask to speed up your data preprocessing. Here is the link 2.) Use cloud computing services like Azure, AWS or GCP. I am not aware of other two but I have worked on Azure and it provides a lot of options for implementing a data science solution. You get options like Auto-ML, Azure Designer, Python ML ...


3

A million observations of 20 features should be very manageable on a laptop, if a little slow. Cloud computing for very large datasets is staggeringly expensive and offers little or no benefit unless and until you have good parallelization in place. I would recommend keeping that option as your last resort. For the initial data exploration and ...


1

Parallelize your analyses on a single (multi-cpu) machine with e.g. pandarallel or the like or go for broke with scala/spark/hadoop if the problem wont fit on a single machine.


Only top voted, non community-wiki answers of a minimum length are eligible