Bidirectional GRU: validation loss stuck on plateau diverges from well performing training loss

Question

tl;dr: What's the interpretation of the validation loss decreasing faster than training loss at first but then get stuck on a plateau earlier and stop decreasing?

The accuracy behaviour is similar.

Background:

The task is multi-class document classification with a high number of labels (L = 48) and a highly imbalanced dataset. It's a NLP task, using only word-embeddings as features.

The model implemented is a Recurrent Neural Network based on Bidirectional GRU layer. The full implementation is as follows:

inp = Input(shape=(MAX_LEN,))
x = Embedding(num_words, embed_size, weights=[embedding_matrix])(inp)
x = SpatialDropout1D(0.2)(x)
x = Bidirectional(GRU(80, return_sequences=True, dropout=0.5, recurrent_dropout=0.5))(x)
avg_pool = GlobalAveragePooling1D()(x)
max_pool = GlobalMaxPooling1D()(x)
conc = concatenate([avg_pool, max_pool])

dense = Dense(num_classes)(conc)
bn = BatchNormalization()(dense)
outp = Activation("softmax")(bn)

adam = Adam(lr=LEARNING_RATE)

model = Model(inputs=inp, outputs=outp)
model.compile(loss=focal_loss,
              optimizer=adam)

Relevant insights:

• Highly regularized: SpatialDropot, Dropout, Recurrent Dropout, Batch normalization...
• Imbalanced sensible loss function: Focal loss
• Adaptative optimizer: Adam

Training setup:

• Epochs: 300
• Learning rate: $1e-4$
• Batch size: 32

Answer 1

Your training and test sets are different. If your test set has less errors than your training set then it means that the data contained in the test set is easier for the algorithm than the data in the train set. For example in the training set you have Shakespeare and in the test set you only have text with short sentences, clear structure and low vocabulary.

The effect that comes after the first phase of learning is called overfitting. The algorithm finds some logic in the data that does not really exist. For example, in the text above word "logic" is always preceded by word "some". But this rule does not really exist and you can still freely say "there is logic" without word "some". You can see this effect in how people think. Imagine you are in a vary dark forest. Your brain will see animals even if there is just a tree or the wind blew some leaves. It is better to see patterns when they do not exist than not see patterns when they exist (or the tiger will eat you). And you can see it in the training. Your model "sees" stuff that does not exist but at the same time still improves pattern recognition that really matters. This goes till epoch 150 in your data where the model starts "seeing" more stuff that does not exist and starts to lose the previously learned patterns. Non realistic patterns, that are only specific to the train set, start to overwhelm the good patterns. This is considered normal for neural networks. Almost all neural networks should stop learning before the training error becomes zero.