0
$\begingroup$

In my effort to learn a bit more about data science I scraped some labeled data from the web and am trying to classify examples into one of three classes. I am running into a problem that, regardless of what model I try, my validation loss flattens out while my training loss continues to decrease (see plot below).

enter image description here

A few potentially relevant notes:

  • I have about 15,000(3,000) training(validation) examples.
  • Labels are roughly evenly distributed and stratified for training and validation sets (class 1: 35%, class 2: 34% class 3: 31%).
  • I have 73 features that consist of: 10 numerical features, 8 categorical features that translate to 43 one-hot encoded features, and a 20-dimensional text embedding
  • I get similar results if I apply PCA to these 73 features (keeping 99% of the variance brings the number of features down to 22).
  • The plot shown here is using XGBoost.XGBClassifier using the metric 'mlogloss', with the following parameters after a RandomizedSearchCV: 'alpha': 7.13, 'lambda': 5.46, 'learning_rate': 0.11, 'max_depth': 7, 'n_estimators': 221
  • I get similar results using a basic Neural Network of Dense and Dropout layers.
  • Best model I've achieved only gets ~66% accuracy on my validation set when classifying examples (and 99% on my training examples).
  • Admittedly my text embedding might not be fantastic (using gensim's fasttext), but they are also the most important feature when I use Xxgboost's plot_importance function. Though, I was facing a similar problem even before I added the text embedding.
$\endgroup$
1
$\begingroup$

This is totally normal and reflects a fundamental phenomenon in data science: overfitting. When the validation loss stops decreasing, while the training loss continues to decrease, your model starts overfitting. This means that the model starts sticking too much to the training set and looses its generalization power. As an example, the model might learn the noise present in the training set as if it was a relevant feature.

When training your model, you should monitor the validation loss and stop the training when the validation loss ceases decreasing significantly. It is also the validation loss that you should monitor while tuning hyperparameters or comparing different preprocessing strategies.

$\endgroup$
2
  • $\begingroup$ Thank you for the comment. I have tried to address that by implementing early stopping when the validation loss stops decreasing. However, the best accuracy I can achieve when stopping at that point is only 66%. Do you have any suggestions to help with the overfitting? I tried running PCA, adding l1/l2 regularization, and reducing the number of features to no avail. At this point is it better feature engineering that might be more correlated with the labels? (note: I cannot acquire more data as I have scraped it all) $\endgroup$
    – Marty
    Nov 17 '20 at 19:27
  • $\begingroup$ Increasing the validation score is the core of the whole work... and maybe the main difficulty! There are several tracks you can explore. You could inspect the false positives and negatives (plot data points, distributions, decision boundary..) and try to understand what the algo misses. You could try other algorithms and see if they perform better. You could try to augment your dataset by generating synthetic data points... $\endgroup$
    – A Co
    Nov 20 '20 at 11:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.