0
$\begingroup$

what does it mean if in a neural network, the training and validation losses are low but the predictions (so use model on test set) are bad?

Neural network architecture:

model = tf.keras.Sequential()
model.add(Masking(mask_value=0., input_shape=(timesteps, features)))
model.add(Bidirectional(LSTM(units=100, return_sequences=True, dropout=0.5, recurrent_dropout=0.5), input_shape=(timesteps, features)))
model.add(Dense(1, activation='sigmoid'))

My loss function is:

def get_top_one_probability(vector):
  return (K.exp(vector) / K.sum(K.exp(vector)))

def listnet_loss(real_labels, predicted_labels):
  return -K.sum(get_top_one_probability(real_labels) * tf.math.log(get_top_one_probability(predicted_labels)))

Fitting settings:

model.fit(training_dataset, training_dataset_labels, validation_split=0.2, batch_size=1, 
            epochs=number_of_epochs, workers=10, verbose=1, 
            callbacks=[SaveModelCallback(), keras.callbacks.EarlyStopping(monitor='val_loss', mode='auto', patience=3)])

Thanks in advance.

$\endgroup$
1
$\begingroup$

Perhabs your network is overfitting. Overfitting is where networks tuned its parameters perfectly to your training data and therefore it has very low loss on training set. Unfortunately, it will perform badly when new samples are provided within test set. Overfitting is broadly descipted almost everywhere: https://en.wikipedia.org/wiki/Overfitting

What you can do with overfitting:

  • try neural network with simplier structure, it should help your network to preserve ability to generalize knowledge,
  • use early stopping; try to measure validation loss at every epoch. And when it gets higher for like 3 epochs in a row - stop network training,
  • dropout: dropout is simple technique that prevents big networks from overfitting by dropping certains connection in each epochs training then averaging results. It is easy to use because it is implemented in many libraries like Keras or PyTorch.
  • $l_1$ and $l_2$ regularization: they are modification to your loss function making it involve sum of weights. Minimizing sum of net's weights prevents situation when network is oversensitive to particular inputs.

The other cause for this situation could be bas data division into training, validation and test set. Training and validation set's loss is low - perhabs they are pretty similiar or correlated, so loss function decreases for both of them. Then relation you try to find could by badly represented by samples in training set and it is fit badly. I would check that division too.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for you answer. $\endgroup$ – pairon Apr 14 at 20:23
  • $\begingroup$ I read better now, sorry. I took 20% of my training set as validation set. I also added dropout and recurrent_dropout to my LSTM layer. NN architecture posted. $\endgroup$ – pairon Apr 14 at 22:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.