New answers tagged


This is a VERY good question. I will break it into 2 parts: Reading and pre-processing: At that scale, libraries like Pandas are usually not a good bet. A simple pd.read_csv could result in an out-of-memory error. Options are: Datatables: - Can read and process large datasets quickly and efficiently DASK(


A million observations of 20 features should be very manageable on a laptop, if a little slow. Cloud computing for very large datasets is staggeringly expensive and offers little or no benefit unless and until you have good parallelization in place. I would recommend keeping that option as your last resort. For the initial data exploration and ...


Parallelize your analyses on a single (multi-cpu) machine with e.g. pandarallel or the like or go for broke with scala/spark/hadoop if the problem wont fit on a single machine.


There are 2 things you can do here: 1.) Use libraries like Dask to speed up your data preprocessing. Here is the link 2.) Use cloud computing services like Azure, AWS or GCP. I am not aware of other two but I have worked on Azure and it provides a lot of options for implementing a data science solution. You get options like Auto-ML, Azure Designer, Python ML ...


Updated Instructions to install Conda on Google Colab Oct 2021 The process is much simpler with condacolab python library Steps Import condacolab python library Install condacolab !pip install -q condacolab import condacolab condacolab.install() Post kernel restart, check condacolab installation import condacolab condacolab.check() Environment You can ...

Top 50 recent answers are included