In my effort to learn a bit more about data science I scraped some labeled data from the web and am trying to classify examples into one of three classes. I am running into a problem that, regardless of what model I try, my validation loss flattens out while my training loss continues to decrease (see plot below).
A few potentially relevant notes:
- I have about 15,000(3,000) training(validation) examples.
- Labels are roughly evenly distributed and stratified for training and validation sets (class 1: 35%, class 2: 34% class 3: 31%).
- I have 73 features that consist of: 10 numerical features, 8 categorical features that translate to 43 one-hot encoded features, and a 20-dimensional text embedding
- I get similar results if I apply PCA to these 73 features (keeping 99% of the variance brings the number of features down to 22).
- The plot shown here is using XGBoost.XGBClassifier using the metric 'mlogloss', with the following parameters after a RandomizedSearchCV: 'alpha': 7.13, 'lambda': 5.46, 'learning_rate': 0.11, 'max_depth': 7, 'n_estimators': 221
- I get similar results using a basic Neural Network of Dense and Dropout layers.
- Best model I've achieved only gets ~66% accuracy on my validation set when classifying examples (and 99% on my training examples).
- Admittedly my text embedding might not be fantastic (using gensim's fasttext), but they are also the most important feature when I use Xxgboost's plot_importance function. Though, I was facing a similar problem even before I added the text embedding.