I have a 400GB data set that I want to train a model on. What is the cheapest method to train this model? The options I can think of so far are:

  1. AWS instance with massive RAM and train CPU (slow, but instances are cheap).
  2. AWS instance with many GPUs and using Dask + XGBoost to distribute (fast, but expensive instances, and I don't even think there would be an instance large enough to handle).

I have just assumed XGBoost is the best package since its tabular data, but if another gradient boosted tree package would be better at handling this, that would be acceptable as well.

Any help would be greatly appreciated.

  • 3
    $\begingroup$ RandomForest and some other linear models allow you to set a parameter named warm_start, in which you can train the model using batches of your data and then retrain it with subsequent batches but the model will use the parameters already learned (Trees already created in the case of RF) All this in Scikit-learn. I think XGB should have an equivalent $\endgroup$ Mar 18 at 2:30
  • $\begingroup$ Linear regression is the fastest and cheapest $\endgroup$ Mar 18 at 9:04
  • $\begingroup$ Sorry I should have specified. I'd like to use gradient boosting like XGBoost or LightGBM or some other gradient boosting package. $\endgroup$
    – lara_toff
    Mar 18 at 10:01
  • $\begingroup$ Fair question. Do you have any reason why it would be necessary to train using the whole data? I mean depending on the structure of your data (time-indexed, user index) You can sample your data from ~ 15% - 25% and use that sample to train. Just considerer if you need to stratify but I do not see any theoretical reason not to sample $\endgroup$ Mar 18 at 15:12
  • $\begingroup$ The model performs noticeably better the more data. Even at 100m rows, there are still gains when adding additional data. $\endgroup$
    – lara_toff
    Mar 22 at 13:55

I don't know many boosting packages but I've been using XGboost for a while now and the biggest tabular dataset I've had was more than 40 times smaller than yours. The training took 2-3 days.

In my experience training time is worse than linear with the size of the data even though it highly depends on the data itself and the hyperparameters you chose. My guess is your training would be very (too) long.

If you really want to use XGboost, you should train on GPU, it seems to me that you are looking at cloud providers, I know google offers managed training of XGboost on GPU, others surely do as well.

With that amount of data I think you should consider using deep learning. You could maybe use tabnet which is a great model developed for tabular data by Google AI. It is easy to try using pytorch for instance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.