Configuring Incremental XGBoost model

Question

I have a large dataset which can't be loaded in memory, hence I decided to use incremental learning using Xgboost. What I have done currently is:

1. Tuned num_boosting_rounds using a chunk of data
2. Set early stopping rounds to a value (60) << num_boosting_rounds to prevent overfitting

When my training code runs, it runs for a 1000 rounds on the first two chunks optimizing the loss function. It then stops at 60 rounds for each subsequent chunk as the best value for loss function was observed in the 1000th round in the 2nd Chunk. Is this the correct way to configure the Incremental model? Would this result in my model being sub-par owing to early stop for a majority of training chunks.

for idx,df in enumerate(df_pointer):
    num_round = 1000
    early_stopping_rounds = 60
    param = {'max_depth':5, 'eta':0.02, 'silent':1, 'objective':'binary:logistic', 'eval_metric':'logloss', 'max_delta_step':4, 'scale_pos_weight': 4}
    dtrain, deval = getDMatrixSplit(df)
    watchlist  = [(deval,'eval')]
    bst = xgb.train(param, dtrain, num_round, watchlist,
    early_stopping_rounds=early_stopping_rounds, xgb_model=xgb_model)
    xgb_model = self.model_path +'/xgb_%s_%s.model'%(ml_algo, idx)
    bst.save_model(xgb_model)

Answer 1

The "optimal" stopping point for XGBoost really depends on the data you feed into it.

Using Chunk N and Chunk N+1 for instance, consider the two scenarios:

1. data in Chunk N and Chunk N+1 is very different - it's likely that the optimal stopping points for each chunk are very different.
2. data in Chunk N and Chunk N+1 is very similar - it's likely that the optimal stopping points for each chunk are quite similar.

I would suggest using a different approach - so for each chunk:

• use the xgb.cv function to first determine the optimal stopping point
• use the results from xgb.cv in xgb.train [i.e. set nrounds = xgb.cv$best.iteration]

As a side-note, have you explored any other techniques / tools to handle large data sets? In R, there are packages designed to tackle similar problems - e.g.bigmemory and ff.