I am creating a RNN in Keras. It was suggested that I utilize a warm-up period before loss is calculated to increase accuracy down the line.

I saw some people achieved this by creating a loss function via tensorflow.

Does Keras have any simple way to directly add a warm-up period?


1 Answer 1


To my knowledge it is currently not possible to compile a Keras model first with a warmup loss function and later re-compile it with another. That would be the cleanest solution in my opinion.

You can however, with a little use of tensorflow (assuming you are using that as a backend judging from your tags) do something along these lines:


def loss(y_true, y_pred):
   seen = tf.Variable(0.)
   seen = tf.assign_add(seen, 1.)

   loss = tf.cond(tf.less(seen, WARMUP_ROUNDS),
                  lambda: warmup_loss(y_true, y_pred),
                  lambda: actual_loss(y_true, y_pred))

   return loss

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.