I am trying to train a LSTM model. Is this model suffering from overfitting?
Here is train and validation loss graph:
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
The model is overfitting right from epoch 10, the validation loss is increasing while the training loss is decreasing.
Dealing with such a Model:
There are many other options as well to reduce overfitting, assuming you are using Keras, visit this link.
Yes this is an overfitting problem since your curve shows point of inflection. This is a sign of very large number of epochs. In this case, model could be stopped at point of inflection or the number of training examples could be increased.
Also, Overfitting is also caused by a deep model over training data. In that case, you'll observe divergence in loss between val and train very early.
Another possible cause of overfitting is improper data augmentation. If you're augmenting then make sure it's really doing what you expect.
I had a similar problem, and it turned out to be due to a bug in my Tensorflow data pipeline where I was augmenting before caching:
def get_dataset(inputfile, batchsize): # Load the data into a TensorFlow dataset. signals, labels = read_data_from_file(inputfile) dataset = tf.data.Dataset.from_tensor_slices((signals, labels)) # Augment the data by dynamically tweaking each training sample on the fly. dataset = dataset.map( map_func=(lambda signals, labels: (tuple(tf.py_function(func=augment, inp=[signals], Tout=[tf.float32])), labels))) # Oops! Should have called cache() before augmenting dataset = dataset.cache() dataset = ... # Shuffle, repeat, batch, etc. return dataset training_data = get_dataset("training.txt", 32) val_data = //... model.fit(training_data, validation_data=val_data, ...)
As a result, the training data was only being augmented for the first epoch, but the validation data was being augmented on every epoch. This caused the model to quickly overfit on the training data while the validation loss continually increased. Moving the augment call after cache() solved the problem.
It's not severe overfitting. So, here is my suggestions:
1- Simplify your network! Maybe your network is too complex for your data. If you have a small dataset or features are easy to detect, you don't need a deep network.
2- Add Dropout layers.
3- Use weight regularization. Here is the link for further information: https://keras.io/api/layers/regularizers/
I had this issue - while training loss was decreasing, the validation loss was not decreasing. I checked and found while I was using LSTM:
(-1,1), I choose
(0,1), this right there reduced my validation loss by the magnitude of one order