I am building another XGBoost model and I'm really trying not to overfit the data. I split my data into train and test set and fit the model with early stopping based on the test-set error which results in the following loss plot:
I'd say this is pretty standard plot with boosting algorithms as XGBoost. My reasoning is that my point of interest is mostly the performance on the test set and until the XGBoost stopped training around 600th epoch due to early stopping the test-set loss was still decreasing. On the other way, overfitting is sometimes defined as a situation when train error decreases faster than test error and this is exactly what happens here. But my intuition is that decision-tree based techniques always drill down the training data (for example, trees in random forest kind of deliberately overfit the training set and it's a role of bagging to reduce the variance). I believe gradient boosting techniques are also known from drilling down the training dataset pretty deep and even with low learning rate we can't help it.
But I may be wrong. Therefore, I'd like to confirm that this situation is not really something to worry about and I do not overfit the data. I'd also like to ask what the perfect learning curves plot would look like with gradient boosting techniques?