Can a gradient boosting solution like XGBoost or Lightbgm be used for a huge amount of data ? I have a csv file of 820GB containing 1 Billion observations and each observation has 650 datapoints.

Is this too much data for XGBoost ? I have searched all over the internet for a solution to when the data won't fit into RAM memory to no avail. I read about external memory for xgb but there is no detailed doc. Can someone point me in the right direction please and thank you !

  • $\begingroup$ exactly what I also want to know for a while with reference to LightGBM. The only thing I know so far is, that you can use CatBoost to "learn from file". But they have no streaming tool yet. Maybe this post about reading data helps towardsdatascience.com/… $\endgroup$
    – Peter
    Jun 15 '19 at 10:06
  • $\begingroup$ Thank for the link, but it's strange there is little info on how to do gradient boosting on massive datasets. Would be super limited if it only works if you can put the data into memory $\endgroup$
    – Medz Benz
    Jun 15 '19 at 17:56
  • $\begingroup$ I think Apache Spark has some extensions/implementations that might be of interest: github.com/Azure/mmlspark/blob/master/docs/lightgbm.md xgboost.readthedocs.io/en/latest/jvm/… spark.apache.org/docs/2.2.0/ml-classification-regression.html Otherwise, is it absolutely necessary to train using all one billion observations? Depending on how difficult your problem is, surely the learning curve tails off very badly after a certain amount of observations? Like a million maybe, or even less? $\endgroup$
    – aranglol
    Jun 15 '19 at 22:31

1) Split data in smaller blocks

2) learn clf on first block

3) Dump model into pickle

4) load second block

5) load model from pickle

6) learn clf on second block and so on


You can try to split you dataset into 100 smaller datasets and build independent models After that generate new dataset with 100 features where each feature is a prediction of some model from 100. Train the final model using semi-independent predictors

At least it will allow you to decrease computation time significantly


You can use Vaex and Koalas with Sparkling water from H2O. All the three combined if you have computation infra. You can process than pretty much very distributed with ease


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.