Why might my validation loss flatten out while my training loss continues to decrease?

Question

In my effort to learn a bit more about data science I scraped some labeled data from the web and am trying to classify examples into one of three classes. I am running into a problem that, regardless of what model I try, my validation loss flattens out while my training loss continues to decrease (see plot below).

A few potentially relevant notes:

• I have about 15,000(3,000) training(validation) examples.
• Labels are roughly evenly distributed and stratified for training and validation sets (class 1: 35%, class 2: 34% class 3: 31%).
• I have 73 features that consist of: 10 numerical features, 8 categorical features that translate to 43 one-hot encoded features, and a 20-dimensional text embedding
• I get similar results if I apply PCA to these 73 features (keeping 99% of the variance brings the number of features down to 22).
• The plot shown here is using XGBoost.XGBClassifier using the metric 'mlogloss', with the following parameters after a RandomizedSearchCV: 'alpha': 7.13, 'lambda': 5.46, 'learning_rate': 0.11, 'max_depth': 7, 'n_estimators': 221
• I get similar results using a basic Neural Network of Dense and Dropout layers.
• Best model I've achieved only gets ~66% accuracy on my validation set when classifying examples (and 99% on my training examples).
• Admittedly my text embedding might not be fantastic (using gensim's fasttext), but they are also the most important feature when I use Xxgboost's plot_importance function. Though, I was facing a similar problem even before I added the text embedding.

Answer 1

This is totally normal and reflects a fundamental phenomenon in data science: overfitting. When the validation loss stops decreasing, while the training loss continues to decrease, your model starts overfitting. This means that the model starts sticking too much to the training set and looses its generalization power. As an example, the model might learn the noise present in the training set as if it was a relevant feature.

When training your model, you should monitor the validation loss and stop the training when the validation loss ceases decreasing significantly. It is also the validation loss that you should monitor while tuning hyperparameters or comparing different preprocessing strategies.