What are the best practices to train xgboost (eXtreme gradient boosting) models on data that is to big to hold it in memory at once? Splitting the data and train multiple models? Are there more elegant solutions?

  • $\begingroup$ Have you tried my suggestion below with the “boost from prediction”? Ironically, I now face similar problem and just wanted to check whether you were able to resolve it somehow. Thank you. $\endgroup$ – aivanov Dec 21 '17 at 21:56
  • $\begingroup$ Does datascience.stackexchange.com/questions/23890/… answer your question? $\endgroup$ – Mortezaaa Dec 29 '17 at 16:27
  • $\begingroup$ Stacking would be a solution for sure. However, to make a prediction in the end you will have to run several models sequentially. It would be more elegant to generate subsets of a large dataset on the fly and build it into a single model. $\endgroup$ – Sören Jan 4 '18 at 8:12

You can train xgboost, calculate the output (margin) and then continue the training, see example in boost from prediction.

I‘ve not tried it myself, but maybe you could train on the first subset of your data (say 10%) and then continue on another subset, etc.


Step by step procedure

  1. Split the data into N manageable subsets, set n=1
  2. Train xgboost on n-th subset
  3. Calculate the prediction (margin) for n+1 subset using the model obtained from previous
  4. Add the margin into the n+1 subset via setinfo
  5. Increment n

Steps 2-5 to be repeated N times.

  • $\begingroup$ i don't think this is what OP is asking. In your link it shows how to pass the WHOLE data through XGBoost and obtain new trees based using the scores from a model that was previous built. The OP is asking how to train the model on A PART of the data and then continue the training on another PART. $\endgroup$ – xiaodai Oct 24 '18 at 3:24

I don't think what you are asking for is possible. See this issue.

I understand that you want to train the model on A PART of the data and then continue the training on another PART and so. So @aivanov's answer will not help in this regard.


If you are using R, have you considered the bigmemory and ff packages?

I don't have much experience using these myself, but would be interested to see if they help with the issue at hand.


Not the answer you're looking for? Browse other questions tagged or ask your own question.