# Low training and validation loss but bad predictions

what does it mean if in a neural network, the training and validation losses are low but the predictions (so use model on test set) are bad?

Neural network architecture:

model = tf.keras.Sequential()
model.add(Bidirectional(LSTM(units=100, return_sequences=True, dropout=0.5, recurrent_dropout=0.5), input_shape=(timesteps, features)))


My loss function is:

def get_top_one_probability(vector):
return (K.exp(vector) / K.sum(K.exp(vector)))

def listnet_loss(real_labels, predicted_labels):
return -K.sum(get_top_one_probability(real_labels) * tf.math.log(get_top_one_probability(predicted_labels)))


Fitting settings:

model.fit(training_dataset, training_dataset_labels, validation_split=0.2, batch_size=1,
epochs=number_of_epochs, workers=10, verbose=1,
callbacks=[SaveModelCallback(), keras.callbacks.EarlyStopping(monitor='val_loss', mode='auto', patience=3)])


Perhabs your network is overfitting. Overfitting is where networks tuned its parameters perfectly to your training data and therefore it has very low loss on training set. Unfortunately, it will perform badly when new samples are provided within test set. Overfitting is broadly descipted almost everywhere: https://en.wikipedia.org/wiki/Overfitting

What you can do with overfitting:

• try neural network with simplier structure, it should help your network to preserve ability to generalize knowledge,
• use early stopping; try to measure validation loss at every epoch. And when it gets higher for like 3 epochs in a row - stop network training,
• dropout: dropout is simple technique that prevents big networks from overfitting by dropping certains connection in each epochs training then averaging results. It is easy to use because it is implemented in many libraries like Keras or PyTorch.
• $$l_1$$ and $$l_2$$ regularization: they are modification to your loss function making it involve sum of weights. Minimizing sum of net's weights prevents situation when network is oversensitive to particular inputs.

The other cause for this situation could be bas data division into training, validation and test set. Training and validation set's loss is low - perhabs they are pretty similiar or correlated, so loss function decreases for both of them. Then relation you try to find could by badly represented by samples in training set and it is fit badly. I would check that division too.

• Thank you for you answer. Apr 14 '20 at 20:23
• I read better now, sorry. I took 20% of my training set as validation set. I also added dropout and recurrent_dropout to my LSTM layer. NN architecture posted. Apr 14 '20 at 22:31