I need to run a Random Forest process with scikit-learn. To train the model, I have a database table with 10 million rows of features. The question is: what is the best way to approach this, should I load into memory the 10 million rows, for example with numpy or pandas or there's a better way to load the data progressively by chunks?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ stackoverflow.com/questions/23872942/sklearn-and-large-datasets $\endgroup$– TitoOrtNov 2, 2020 at 22:51
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
There are multiple possiblities from dusk, to others model etc.
Here are my 2 favorites, not to loose you in the number of possibilities:
www.h5py.org/ "It lets you store huge amounts of numerical data, and easily manipulate that data from NumPy. For example, you can slice into multi-terabyte datasets stored on disk, as if they were real NumPy arrays. Thousands of datasets can be stored in a single file, categorized and tagged however you want."
Try online learning with Cousin models of random forest (light-gbm). He has online learning capabilities.
