Both train and test error are decreasing in XGBoost iterations

Question

I have an issue with training an XGBoost classifier in a sence that both train and test error only decrease throughout more iterations (num_boost_round) even if I use 1000 num boost rounds and 10 early stopping rounds. Then when I try to apply the model on a separate set not used for testing and training I can see that a model trained over 100 rounds performs much better than 1000 one. My learning rate is 0.01.

I am wondering whether this is a normal situation (and I should just do early stopping on this independent validation set) or whether I am doing something wrong. Theoretically, the test error should start increasing at some point, but this is just not happening.

I am using XGBoost and I am splitting my dataset in train/test like this:

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.3, random_state=42)

dtrain = xgb.DMatrix(X_train, label=y_train)
dtest = xgb.DMatrix(X_test, label=y_test)

Them I train it in the following way:

watchlist =  [(dtest, 'eval'), (dtrain, 'train')]
progress = dict()

# Train and predict with early stopping
xg_reg = xgb.train(
    params=params,
    dtrain=dtrain, num_boost_round=boost_rounds,
    evals=watchlist,    # using validation on a test set for early stopping; ideally should be a separate validation set
    early_stopping_rounds=early_stopping,
    evals_result=progress)

Note that I have used gridsearch to find the optimal hyperparameters (params).

A plot which I get (yellow is my cutoff point identified pretty much via testing on the independent set):

Answer 1

This is a common situation and can be caused by a number of things, such as:

1. Too much variance in the model (e.g. did the hyperparameter tuning identify low values for eta, alpha, lambda, gamma, subsampling/column sampling/row sampling, minimum child weights and/or high values for depth and number of leaves?)
2. Having a small dataset with high variance. XGBoost is prone to overfitting and this is exacerbated when there isn't much data for it.
3. Not shuffling the data so that the validation set has lower variance than (or differs in distribution from) the training and test sets. train_test_split shuffles by default, so this shouldn't be a problem if you've used it as indicated in the question for generating the validation set also.

Stopping training early is a perfectly valid and common method of increasing bias in a model and doing so based on a validation set is acceptable. It is preferable for the training to stop based on the test set. To achieve this you could address the potential issues above. Without knowing more about the data and how the validation set was generated, the most reliable advice I can give is to alter the hyperparameters to increase model bias.

Answer 2

Simply you are using the training set for early stopping.

Try by setting

watchlist =  [(dtrain, 'train'), (dtest, 'eval')] # the last one is used for early stopping