0
$\begingroup$

I'm training an LSTM with Keras. I notice that the loss value decreases, but very slowly.

11748/11748 [==============================] - 71s 6ms/step - loss: 0.4564 - val_loss: 0.4543
11748/11748 [==============================] - 71s 6ms/step - loss: 0.4524 - val_loss: 0.4519
11748/11748 [==============================] - 71s 6ms/step - loss: 0.4517 - val_loss: 0.4515
11748/11748 [==============================] - 71s 6ms/step - loss: 0.4514 - val_loss: 0.4513
11748/11748 [==============================] - 75s 6ms/step - loss: 0.4512 - val_loss: 0.4511
11748/11748 [==============================] - 72s 6ms/step - loss: 0.4511 - val_loss: 0.4510

My model:

# Model definition
model = tf.keras.Sequential()
model.add(Masking(mask_value=0., input_shape=(timesteps, features)))
model.add(Bidirectional(LSTM(units=100, return_sequences=True), input_shape=(timesteps, features)))
model.add(Dense(1, activation='sigmoid'))

model.summary()
model.compile(loss=listnet_loss, optimizer=keras.optimizers.Adam(learning_rate=0.00005, beta_1=0.9, beta_2=0.999, amsgrad=False))

My loss function:

def get_top_one_probability(vector):
  return (K.exp(vector) / K.sum(K.exp(vector)))

def listnet_loss(real_labels, predicted_labels):
  return -K.sum(get_top_one_probability(real_labels) * tf.math.log(get_top_one_probability(predicted_labels)))

I'm also standardizing my training set using the StandardScaler from scikit-learn.

Is the problem associated with the optimizer or the learning rate?

$\endgroup$
6
  • 1
    $\begingroup$ Your learning rate is very low, try increasing it to increase the loss rate. $\endgroup$ – bkshi Apr 16 '20 at 15:55
  • $\begingroup$ Try to check Gradient distributions to know whether you have any vanishing gradient problem. $\endgroup$ – Uday Apr 16 '20 at 16:47
  • $\begingroup$ @Uday how could I do this? $\endgroup$ – pairon Apr 16 '20 at 16:48
  • $\begingroup$ @bkshi thank you for the suggestion. $\endgroup$ – pairon Apr 16 '20 at 16:49
  • $\begingroup$ You can use tensorboard for the gradient histograms. check the tensorboard callback or tf.summaries $\endgroup$ – Uday Apr 16 '20 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.