1
$\begingroup$

I'm working on a classification problem with a very large dataset (a little under 1 billion obs) and around 25 predictors.

I'm doing this analysis in R on a VM with 128GB of memory, but am still hitting memory issues when training certain types of models, not to mention having to wait for quite a while for runs to complete. I'm primarily using logistic regression and random forests, and keeping my training dataset to 10% of the overall sample.

What are some solutions (packages, platforms, techniques) I could use to address these memory and/or speed issues? I'm fairly comfortable with Python, so solutions do not have to be R-specific.

Improve this question
$\endgroup$
5
  • $\begingroup$ 10% of 1 trillion is still 100 billion. Do you really need such a big training set? I guess for experimental purposes you can limit yourself to 1000 samples an this would do the job for you. Once confident about the approach, you can try 10,000 or even 100,000 for a training set, but I can hardly see the case of needing more. Do you have any particular reasons to want a bigger training set? $\endgroup$
    – mapto
    Commented Aug 3, 2018 at 15:18
  • $\begingroup$ Whoops, meant to say 1 billion sorry. I've updated the post. Part of the reason for needing larger samples is that the class I want to predict is very rare; only around 0.3% of the entire sample. $\endgroup$
    – MrT
    Commented Aug 3, 2018 at 15:28
  • 2
    $\begingroup$ How about using some sort of representative sampling (stratified or cluster) that would make sure to have a good representation of both positive and negative examples? I'd still suggest that early experimentation is much more efficient with much smaller data set, just make sure to boost the share of your positive examples. $\endgroup$
    – mapto
    Commented Aug 3, 2018 at 15:38
  • $\begingroup$ Hmm, that's a good idea. I will need to figure out the appropriate sampling approach, so as not to introduce bias. $\endgroup$
    – MrT
    Commented Aug 3, 2018 at 15:59
  • $\begingroup$ In my experience, boosted tree techniques are faster and produce better results than random forests. I'd suggest using tools like xgboost, lightgbm, and catboost on your smaller stratified sample. $\endgroup$
    – bradS
    Commented Sep 6, 2018 at 7:56

2 Answers 2

Reset to default
1
$\begingroup$

LightGBM is the answer you are looking for. It uses less memory than xgboost, catboost and the fastest GBM library. Also it has a RF mode which is faster than sklearn implementation. It works on both R and Python. Also it has some parameters to control memory management.

You can check this benchmark repo.

Improve this answer
$\endgroup$
1
  • $\begingroup$ LightGBM is definitely the state of art answer for that. Its booster Gradient One-Sided Sampling (GOSS), the way it deals with categorical features as Effective Feature Bundling (EFB) makes it currently the perfect algorithm for such a large dataset. Moreover, it has a depth-wise tree algorithm unlike XGBoost, which also provides time-efficiency. Be sure that your features are categorical in general and digitize them if they are not to be abla to use the power of EFB, it has such a fascinating way of encoding them and dealing with the sparsity. $\endgroup$
    – Ugur MULUK
    Commented Nov 8, 2018 at 19:24
0
$\begingroup$

You could try a bagging approach, by training the model separately on many random sub-samples and averaging the results, which might ultimately improve your prediction anyway.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.