1
$\begingroup$

I need to run a Random Forest process with scikit-learn. To train the model, I have a database table with 10 million rows of features. The question is: what is the best way to approach this, should I load into memory the 10 million rows, for example with numpy or pandas or there's a better way to load the data progressively by chunks?

$\endgroup$
1
3
$\begingroup$

There are multiple possiblities from dusk, to others model etc.

Here are my 2 favorites, not to loose you in the number of possibilities:

  1. www.h5py.org/ "It lets you store huge amounts of numerical data, and easily manipulate that data from NumPy. For example, you can slice into multi-terabyte datasets stored on disk, as if they were real NumPy arrays. Thousands of datasets can be stored in a single file, categorized and tagged however you want."

  2. Try online learning with Cousin models of random forest (light-gbm). He has online learning capabilities.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.