7
$\begingroup$

I have a 400GB data set that I want to train a model on. What is the cheapest method to train this model? The options I can think of so far are:

  1. AWS instance with massive RAM and train CPU (slow, but instances are cheap).
  2. AWS instance with many GPUs and using Dask + XGBoost to distribute (fast, but expensive instances, and I don't even think there would be an instance large enough to handle).

I have just assumed XGBoost is the best package since its tabular data, but if another gradient boosted tree package would be better at handling this, that would be acceptable as well.

Any help would be greatly appreciated.

Improve this question
$\endgroup$
5
  • 3
    $\begingroup$ RandomForest and some other linear models allow you to set a parameter named warm_start, in which you can train the model using batches of your data and then retrain it with subsequent batches but the model will use the parameters already learned (Trees already created in the case of RF) All this in Scikit-learn. I think XGB should have an equivalent $\endgroup$
    – Multivac
    Commented Mar 18, 2021 at 2:30
  • $\begingroup$ Linear regression is the fastest and cheapest $\endgroup$
    – Carlos Mougan
    Commented Mar 18, 2021 at 9:04
  • $\begingroup$ Sorry I should have specified. I'd like to use gradient boosting like XGBoost or LightGBM or some other gradient boosting package. $\endgroup$
    – lara_toff
    Commented Mar 18, 2021 at 10:01
  • 1
    $\begingroup$ Fair question. Do you have any reason why it would be necessary to train using the whole data? I mean depending on the structure of your data (time-indexed, user index) You can sample your data from ~ 15% - 25% and use that sample to train. Just considerer if you need to stratify but I do not see any theoretical reason not to sample $\endgroup$
    – Multivac
    Commented Mar 18, 2021 at 15:12
  • 1
    $\begingroup$ The model performs noticeably better the more data. Even at 100m rows, there are still gains when adding additional data. $\endgroup$
    – lara_toff
    Commented Mar 22, 2021 at 13:55

2 Answers 2

Reset to default
3
$\begingroup$

Yes, you can train XGBoost in parallel using the Dask backend.

Short Solution

Training XGBoost in parallel with Dask requires 2 changes in your code:

  1. substitute dtrain = xgb.DMatrix(X_train, y_train) with dtrain = xgb.dask.DaskDMatrix(X_train, y_train)

  2. substitute xgb.train(params, dtrain, ...) with xgb.dask.train(client, params, dtrain, ...)

Have a look at this tutorial for a step-by-step guide that trains XGBoost on 100GB in under 4mins. Disclaimer: I work for Coiled, a paid service that offers managed Dask clusters.

Improve this answer
$\endgroup$
1
$\begingroup$

I don't know many boosting packages but I've been using XGboost for a while now and the biggest tabular dataset I've had was more than 40 times smaller than yours. The training took 2-3 days.

In my experience training time is worse than linear with the size of the data even though it highly depends on the data itself and the hyperparameters you chose. My guess is your training would be very (too) long.

If you really want to use XGboost, you should train on GPU, it seems to me that you are looking at cloud providers, I know google offers managed training of XGboost on GPU, others surely do as well.

With that amount of data I think you should consider using deep learning. You could maybe use tabnet which is a great model developed for tabular data by Google AI. It is easy to try using pytorch for instance.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.