I am trying to train an XGBoost model on a quite big dataset (tens of GB, almost a hundred).

I have been trying to use some libraries such as Dask to deal with this problem, without any success due to high memory consumption (I opened a question on Stackoverflow, if you can answer, you are welcome).

Due to the above failure, I decided to implement my own incremental XGBoost training, something like what has been proposed here: XGBoost incremental training

The main idea is basically the following code:

params = {} # your params here
ith_batch = 0
n_batches = 100
model = None
while ith_batch < n_batches:
    d_train = getBatchData(ith_batch)
    model = xgb.train(params, d_train, xgb_model=model)
    ith_batch += 1

However I have three important questions:

  1. Is this method of incremental training the same as training on the whole dataset? If not, why?
  2. How should I split my batches at each iteration? Should I basically sweep the whole dataset sequentially?
  3. How do the most common distributed/parallel compute libraries (Dask-ML, SparkML ecc...) train these models without storing everything in memory? Do they rely on incremental training?
  • $\begingroup$ Spark works in memory, but because it is distributed it utilizes memory of each core. Nevertheless some algorithms can't work in distributed manner. $\endgroup$
    – tkarahan
    May 3, 2021 at 7:15
  • $\begingroup$ @tkarahan Do you have any info on dask_ml? It should be able to deal with big data without loading everything to memory. However, from my many experiments, it seems that it behaves in the same way as Spark ML $\endgroup$ May 3, 2021 at 7:30
  • 1
    $\begingroup$ I have no sufficient info about that, sorry. It's an important tool, in my learning list too. :) $\endgroup$
    – tkarahan
    May 3, 2021 at 8:21
  • $\begingroup$ No problems! Let's wait for someone who knows more about that $\endgroup$ May 3, 2021 at 8:23
  • $\begingroup$ @MattiaSurricchio did you find a solution for that? I am confronted with the same problem. $\endgroup$
    – Herr Derb
    Apr 4, 2023 at 12:42


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.