I have an issue with training an XGBoost classifier in a sence that both train and test error only decrease throughout more iterations (num_boost_round) even if I use 1000 num boost rounds and 10 early stopping rounds. Then when I try to apply the model on a separate set not used for testing and training I can see that a model trained over 100 rounds performs much better than 1000 one. My learning rate is 0.01.
I am wondering whether this is a normal situation (and I should just do early stopping on this independent validation set) or whether I am doing something wrong. Theoretically, the test error should start increasing at some point, but this is just not happening.
I am using XGBoost and I am splitting my dataset in train/test like this:
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.3, random_state=42) dtrain = xgb.DMatrix(X_train, label=y_train) dtest = xgb.DMatrix(X_test, label=y_test)
Them I train it in the following way:
watchlist = [(dtest, 'eval'), (dtrain, 'train')] progress = dict() # Train and predict with early stopping xg_reg = xgb.train( params=params, dtrain=dtrain, num_boost_round=boost_rounds, evals=watchlist, # using validation on a test set for early stopping; ideally should be a separate validation set early_stopping_rounds=early_stopping, evals_result=progress)
Note that I have used gridsearch to find the optimal hyperparameters (params).
A plot which I get (yellow is my cutoff point identified pretty much via testing on the independent set):