0
$\begingroup$

tl;dr: What's the interpretation of the validation loss decreasing faster than training loss at first but then get stuck on a plateau earlier and stop decreasing?

Train vs val loss

The accuracy behaviour is similar.

Background:

The task is multi-class document classification with a high number of labels (L = 48) and a highly imbalanced dataset. It's a NLP task, using only word-embeddings as features.

labels

The model implemented is a Recurrent Neural Network based on Bidirectional GRU layer. The full implementation is as follows:

inp = Input(shape=(MAX_LEN,))
x = Embedding(num_words, embed_size, weights=[embedding_matrix])(inp)
x = SpatialDropout1D(0.2)(x)
x = Bidirectional(GRU(80, return_sequences=True, dropout=0.5, recurrent_dropout=0.5))(x)
avg_pool = GlobalAveragePooling1D()(x)
max_pool = GlobalMaxPooling1D()(x)
conc = concatenate([avg_pool, max_pool])

dense = Dense(num_classes)(conc)
bn = BatchNormalization()(dense)
outp = Activation("softmax")(bn)

adam = Adam(lr=LEARNING_RATE)

model = Model(inputs=inp, outputs=outp)
model.compile(loss=focal_loss,
              optimizer=adam)

Relevant insights:

  • Highly regularized: SpatialDropot, Dropout, Recurrent Dropout, Batch normalization...
  • Imbalanced sensible loss function: Focal loss
  • Adaptative optimizer: Adam

Training setup:

  • Epochs: 300
  • Learning rate: $1e-4$
  • Batch size: 32
$\endgroup$
  • $\begingroup$ Could you plot accuracy for each class and also number of points for each class in train and test separately? $\endgroup$ – keiv.fly Oct 31 '18 at 12:23
  • $\begingroup$ This is the F1-score per class (i.imgur.com/7AmZPh0.png). The number of points for each class in train/test is proportional to the full set, because the splits are made with stratification. Doesn't seem to be strong correlation between F1-Score and support (i.imgur.com/gQqqQRW.png) $\endgroup$ – Alber8295 Nov 1 '18 at 8:34
1
$\begingroup$

Your training and test sets are different. If your test set has less errors than your training set then it means that the data contained in the test set is easier for the algorithm than the data in the train set. For example in the training set you have Shakespeare and in the test set you only have text with short sentences, clear structure and low vocabulary.

The effect that comes after the first phase of learning is called overfitting. The algorithm finds some logic in the data that does not really exist. For example, in the text above word "logic" is always preceded by word "some". But this rule does not really exist and you can still freely say "there is logic" without word "some". You can see this effect in how people think. Imagine you are in a vary dark forest. Your brain will see animals even if there is just a tree or the wind blew some leaves. It is better to see patterns when they do not exist than not see patterns when they exist (or the tiger will eat you). And you can see it in the training. Your model "sees" stuff that does not exist but at the same time still improves pattern recognition that really matters. This goes till epoch 150 in your data where the model starts "seeing" more stuff that does not exist and starts to lose the previously learned patterns. Non realistic patterns, that are only specific to the train set, start to overwhelm the good patterns. This is considered normal for neural networks. Almost all neural networks should stop learning before the training error becomes zero.

$\endgroup$
  • $\begingroup$ My training and test sets are perfectly splitted. A random 70/30 split is done, in every run it appears the same pattern. Moreover, given the imbalance, I'm performing a stratified split. I know what overfitting is and I know that is the graphs tell something like that, but I already have a lot of regularization techniques on. Any advice to counter that? Thanks! $\endgroup$ – Alber8295 Oct 31 '18 at 9:01
  • $\begingroup$ It most likely means you have one (or several) big outliers that are not predictable at all. Plot y_real vs y_pred for the last model that has the same accuracy. Then remove outliers and see if it improves the accuracy. It could be that the model starts to fit better to very few outliers and loses average accuracy. $\endgroup$ – keiv.fly Oct 31 '18 at 9:38
  • $\begingroup$ And look at SGD with momentum. They have better validation accuracy. For example, look how they implement it in ResNet. $\endgroup$ – keiv.fly Oct 31 '18 at 9:42
  • $\begingroup$ The task is document classification, I can't really detect an outlier. I'll try SGD with momentum, but the Adam optimizer also deals with momentum. $\endgroup$ – Alber8295 Oct 31 '18 at 9:59
  • 1
    $\begingroup$ You can detect outliers in classification. Plot losses for each document. I need to find a link about it. The is a paper comparing SGD with Adam and SGD wins. $\endgroup$ – keiv.fly Oct 31 '18 at 10:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.