I'm currently tring to train a model with XGBoost.

My dataset has ~7 million records and 61 columns.

The problem I'm currently having is that I get a MemoryError on python when I try to train the model.

Investigating on the XGboost documentation I can see that I can train a pre trained model, so I thought I could implement a sequential training dividing my data into smaller pieces.

Have anyone already done this? If so, could you please share or recommend the way to do it?

I'm currently using the SkLearn XGBClassifier model.



What you want to do is a sort of manual stacking. Start by training $M_1$ with $n_1$ samples. Then predict the margin for $n_2$ sample with $M_1$ (output_margin=True). Set the margin for $n_2$ samples and train $M_2$ using $n_2$ samples. In round $k$, you need to predict with $M_i$, $i = 1,...,k-1$ using the margins from model $i-1$ and output the margin to model $i+1$.

-- EDIT Pseudocode in R:

model_1 <- xgb.train(data = X1, watchlist = Xn1) 

X2 <- xgb.Dmatrix(X2)
base1 <- predict(model_1, X2, outputmargin=TRUE)
setinfo(X2, "base_margin", base1)

model_2 <- xgb.train(data = X2, watchlist = Xn2) 

X3 <- xgb.Dmatrix(X3)
base1 <- predict(model_1, X3, outputmargin=TRUE)
setinfo(X3, "base_margin", base1)
base2 <- predict(model_2, X3, outputmargin=TRUE)
setinfo(X3, "base_margin", base2)

model_3 <- xgb.train(data = X3, watchlist = Xn3) 
  • $\begingroup$ I'm not quite sure I understand the procedure. Is there a way you could explain the same procedure using pseudo code? That way I can relate the explanation to code. It will help me a lot in understanding this kind of explanation too. $\endgroup$
    – Juanu
    Oct 26 '17 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.