New answers tagged

0

This is a VERY good question. I will break it into 2 parts: Reading and pre-processing: At that scale, libraries like Pandas are usually not a good bet. A simple pd.read_csv could result in an out-of-memory error. Options are: Datatables: https://github.com/h2oai/datatable - Can read and process large datasets quickly and efficiently DASK(https://dask.org/)...


4

A million observations of 20 features should be very manageable on a laptop, if a little slow. Cloud computing for very large datasets is staggeringly expensive and offers little or no benefit unless and until you have good parallelization in place. I would recommend keeping that option as your last resort. For the initial data exploration and ...


1

Parallelize your analyses on a single (multi-cpu) machine with e.g. pandarallel or the like or go for broke with scala/spark/hadoop if the problem wont fit on a single machine.


4

There are 2 things you can do here: 1.) Use libraries like Dask to speed up your data preprocessing. Here is the link 2.) Use cloud computing services like Azure, AWS or GCP. I am not aware of other two but I have worked on Azure and it provides a lot of options for implementing a data science solution. You get options like Auto-ML, Azure Designer, Python ML ...


2

Updated Instructions to install Conda on Google Colab Oct 2021 The process is much simpler with condacolab python library Steps Import condacolab python library Install condacolab !pip install -q condacolab import condacolab condacolab.install() Post kernel restart, check condacolab installation import condacolab condacolab.check() Environment You can ...


Top 50 recent answers are included