Hot answers tagged


There are 2 things you can do here: 1.) Use libraries like Dask to speed up your data preprocessing. Here is the link 2.) Use cloud computing services like Azure, AWS or GCP. I am not aware of other two but I have worked on Azure and it provides a lot of options for implementing a data science solution. You get options like Auto-ML, Azure Designer, Python ML ...


A million observations of 20 features should be very manageable on a laptop, if a little slow. Cloud computing for very large datasets is staggeringly expensive and offers little or no benefit unless and until you have good parallelization in place. I would recommend keeping that option as your last resort. For the initial data exploration and ...


Parallelize your analyses on a single (multi-cpu) machine with e.g. pandarallel or the like or go for broke with scala/spark/hadoop if the problem wont fit on a single machine.

Only top voted, non community-wiki answers of a minimum length are eligible