Can a Warm-up loss period easily be implemented in Keras?

Question

I am creating a RNN in Keras. It was suggested that I utilize a warm-up period before loss is calculated to increase accuracy down the line.

I saw some people achieved this by creating a loss function via tensorflow.

Does Keras have any simple way to directly add a warm-up period?

Answer 1

To my knowledge it is currently not possible to compile a Keras model first with a warmup loss function and later re-compile it with another. That would be the cleanest solution in my opinion.

You can however, with a little use of tensorflow (assuming you are using that as a backend judging from your tags) do something along these lines:

WARMUP_ROUNDS = 10

def loss(y_true, y_pred):
   seen = tf.Variable(0.)
   seen = tf.assign_add(seen, 1.)

   loss = tf.cond(tf.less(seen, WARMUP_ROUNDS),
                  lambda: warmup_loss(y_true, y_pred),
                  lambda: actual_loss(y_true, y_pred))

   return loss