33
$\begingroup$

I am trying to train a LSTM model. Is this model suffering from overfitting?

Here is train and validation loss graph:

loss

$\endgroup$
1
  • 2
    $\begingroup$ All the other answers assume this is an overfitting problem. While it could all be true, this could be a different problem too. Maybe your neural network is not learning at all. It's not possible to conclude with just a one chart. Check the model outputs and see whether it has overfit and if it is not, consider this either a bug or an underfitting-architecture problem or a data problem and work from that point onward. $\endgroup$ Jun 16, 2020 at 3:17

6 Answers 6

40
$\begingroup$

The model is overfitting right from epoch 10, the validation loss is increasing while the training loss is decreasing.

Dealing with such a Model:

  1. Data Preprocessing: Standardizing and Normalizing the data.
  2. Model compelxity: Check if the model is too complex. Add dropout, reduce number of layers or number of neurons in each layer.
  3. Learning Rate and Decay Rate: Reduce the learning rate, a good starting value is usually between 0.0005 to 0.001. Also consider a decay rate of 1e-6.

There are many other options as well to reduce overfitting, assuming you are using Keras, visit this link.

$\endgroup$
1
  • 1
    $\begingroup$ How can we play with learning and decay rates in Keras implementation of LSTM? $\endgroup$ Mar 8, 2019 at 16:33
8
$\begingroup$

Yes this is an overfitting problem since your curve shows point of inflection. This is a sign of very large number of epochs. In this case, model could be stopped at point of inflection or the number of training examples could be increased.

Also, Overfitting is also caused by a deep model over training data. In that case, you'll observe divergence in loss between val and train very early.

$\endgroup$
4
$\begingroup$

I had this issue - while training loss was decreasing, the validation loss was not decreasing. I checked and found while I was using LSTM:

  • I simplified the model - instead of 20 layers, I opted for 8 layers.
  • Instead of scaling within range (-1,1), I choose (0,1), this right there reduced my validation loss by the magnitude of one order
  • I reduced the batch size from 500 to 50 (just trial and error)
  • I added more features, which I thought intuitively would add some new intelligent information to the X->y pair
$\endgroup$
4
$\begingroup$

Another possible cause of overfitting is improper data augmentation. If you're augmenting then make sure it's really doing what you expect.

I had a similar problem, and it turned out to be due to a bug in my Tensorflow data pipeline where I was augmenting before caching:

    def get_dataset(inputfile, batchsize):
        # Load the data into a TensorFlow dataset.
        signals, labels = read_data_from_file(inputfile)
        dataset = tf.data.Dataset.from_tensor_slices((signals, labels))
    
        # Augment the data by dynamically tweaking each training sample on the fly.
        dataset = dataset.map(
                    map_func=(lambda signals, labels: (tuple(tf.py_function(func=augment, inp=[signals], Tout=[tf.float32])), labels)))
    
        # Oops! Should have called cache() before augmenting
        dataset = dataset.cache()
        dataset = ... # Shuffle, repeat, batch, etc.
        return dataset
            
    training_data = get_dataset("training.txt", 32)
    val_data = //...
    
    model.fit(training_data, validation_data=val_data, ...)

As a result, the training data was only being augmented for the first epoch. This caused the model to quickly overfit on the training data. Moving the augment call after cache() solved the problem.

$\endgroup$
3
  • 2
    $\begingroup$ Why would you augment the validation data? $\endgroup$
    – TomSelleck
    Mar 31, 2021 at 10:47
  • 2
    $\begingroup$ @TomSelleck Good catch. I didn't augment the validation data in the real code. I overlooked that when I created this simplified example. Edited my answer so that it doesn't show validation data augmentation. $\endgroup$
    – Kevin D.
    Mar 31, 2021 at 17:29
  • 1
    $\begingroup$ I encountered the same issue too, where the crop size after random cropping is inappropriate (i.e., too small to classify) $\endgroup$
    – Chris Tang
    May 30, 2022 at 13:57
2
$\begingroup$

It may be that you need to feed in more data, as well. If the model overfits, your dataset may be so small that the high capacity of the model makes it easily fit this small dataset, while not delivering out-of-sample performance. In other words, it does not learn a robust representation of the true underlying data distribution, just a representation that fits the training data very well.

Solutions as stated above:

  • reduce model complexity: if you feel your model is not really overly complex, you should try running on a larger dataset, at first.
  • regularization: using dropout and other regularization techniques may assist the model in generalizing better.

I propose to extend your dataset (largely), which will be costly in terms of several aspects obviously, but it will also serve as a form of "regularization" and give you a more confident answer. In case you cannot gather more data, think about clever ways to augment your dataset by applying transforms, adding noise, etc to the input data (or to the network output).

$\endgroup$
1
$\begingroup$

It's not severe overfitting. So, here is my suggestions:

1- Simplify your network! Maybe your network is too complex for your data. If you have a small dataset or features are easy to detect, you don't need a deep network.

2- Add Dropout layers.

3- Use weight regularization. Here is the link for further information: https://keras.io/api/layers/regularizers/

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.