5
$\begingroup$

I'm using xgboost package in R with early stopping at 75 rounds. To monitor the progress the algorithm I print the F1 score from the training and test set after each round. After the algorithm has done 75 rounds, xgboost returns the model with the highest score on the test set, not the training set. My guess is that this has to do with the monitoring functionality and the watchlist parameter of xgboost

My intention of giving the algorithm access to the test set during training (using the watchlist parameter) was to monitor the training progress, and not to select the best performing classifier with respect to the test set. That would be cheating right? My question is two-fold:

  1. Is this really cheating?

and if it is cheating,

  1. Why is it programmed into the xgboost package?
$\endgroup$
  • 2
    $\begingroup$ Call your current test data as validating one. And keep some data as test set separately. Test the best model at the end on this so called never seen slice of test set data. $\endgroup$ – Kiritee Gak Apr 17 '18 at 21:33
1
$\begingroup$

That's not cheating. If your test set is a representative sample of the future data you'll want to make predictions on, you'll want to have the lowest possible error there!

If you maximized performance on the training set, instead, you might overfit. It's fairly easy for a boosted algorithm to inadvertently memorize its training data rather than learn a meaningful mapping of inputs to output.

Per the comment below, the "test set" you describe is actually functioning like a validation set here.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I do not agree. The number of rounds is a parameter to be chosen by cross-validation, a validation set or black magic - but definitively not by a test data set. $\endgroup$ – Michael M Apr 18 '18 at 5:07
  • $\begingroup$ I think here the "test set" the asker describing is acting like the "validation set" you're describing. $\endgroup$ – Eduard Gelman Apr 18 '18 at 10:22
  • $\begingroup$ Exactly. My test set was acting as a validation set which is incorrect. I was not aware of the difference between validation and test set before. $\endgroup$ – kog Apr 18 '18 at 22:23
1
$\begingroup$

The early stopping and watchlist parameters in xgboost can be used to prevent overfitting. If the watchlist is given two data-sets, then the algorithm will perform hold out validation as described here.

Putting the test set in the watchlist will cause the algorithm to select the model with the best performance against the test set which can be considered as cheating.

I was calling my algorithm like this

train <- data[selection,]
test <- data[-selection,]

watchlist <- list(train=train,test=test)

bstSparse <- xgb.train(data=train, 
                       ...,
                       watchlist=watchlist,
                       early_stopping_rounds=75)

But I should have been calling it like this

train_and_val <- data[selection,]
inds <- createDataPartition(temp_and_val$labels, p = 0.9)

train <- train_and_val[inds,]
validation <- train_and_val[-inds,]
test <- data[-selection,]

watchlist <- list(validation=validation,train=train)

bstSparse <- xgb.train(data=train, 
                       ...,
                       watchlist=watchlist,
                       early_stopping_rounds=75)

I found this article very informative with regards to early stopping and xgboost

XGBoost Validation and Early Stopping in R

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.