1
$\begingroup$

I trained a neural network model, a MLP type of network, where the first several layers are 1-D convolution for processing sequence type of input.

However, the training process looks like as follows, where the orange line represents the validation loss and the blue line represents the training loss. The validation loss is large compared to the training loss and the training loss also stops decreasing after the first several iterations. Are there any generic guidance to improve the performance? I have about 1 million training traces, and the number of parameters of the network is about 140K.

enter image description here

$\endgroup$
  • $\begingroup$ Are you sure the validation set is drawn from the same distribution as your training set? You should shuffle your set before splitting. $\endgroup$ – JahKnows May 15 '18 at 10:28
1
$\begingroup$

When the training loss is lower than the validation loss, the model is said to overfit the training data, i.e. it has learned so much from the training data that it only adjusts well to it and it can't generalize to new data. This phenomena is regarded as the variance of the model. The bias of the model is the difference between the training loss and and the loss you've previoussly selected as the minimum loss reachable, or the desired one.

However, this analysts is usually done over other well known metrics, such as precision and recall. You first calculate these metrics on your training data, and then, on the evaluation data. Then you perform the analysis taking the same considerations.

In order to reduce the variance/overfitting, there are common techniques:

  1. Increse the training data by adding more instances, if available
  2. If no more instances are available, perform data augmentation to increase the training dataset
  3. Use regularization, for example, dropout.
  4. Shorten the network. If there are lots of layers, the network may be learning too specific features from the training data

As mentioned before, I'd perform the analysis using other metrics rather than the loss.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Excellent answer. can you please give some insight on data augmentation? $\endgroup$ – deepguy May 15 '18 at 15:57
  • $\begingroup$ Data augmentation is the process by which synthetic data is generated from the original data. In essence, with data augmentation, modified replicas of the original database are generated in order to include more variability. For instance, in image classification, from one image you can generate rotated images, blurred images, scaled images... The key aspect of data augmentation is that these modified replicas should be representative of the population from which the database comes. In other words, you try to include extra instances that can appear in the future in a real application $\endgroup$ – ignatius May 16 '18 at 7:44
0
$\begingroup$

This is clearly a case of overfitting, as your validation loss is much higher than your training loss. I would proceed by performing dropout or weight decay: both of them are regularizers. The regularization technique aim is to set the orange line closer to the blue line. If this works, then you have to ask yourself: is the blue line where I aim to? Or do I want a smaller loss? If you want a smaller loss, you can try using another optimizer, another learning rate or training through more epochs (your training error is still decreasing). However, your first move should be using a regularization technique.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.