In Tensorflow documentation, it is shown how to tune several hyperparameters but not the learning rate.I have searched how to tune learning rate using HParams dashboard but could not find much. The only example is another question on github but it does not work.Can you please give me some suggestions on this?Should I use a callback function?Or provide different learning rates in hp_optimizer as in the question in github? Or something else?

Parts of my code is below:

HP_NUM_UNITS = hp.HParam('num_units', hp.Discrete([16, 32]))
HP_OPTIMIZER = hp.HParam('optimizer', hp.Discrete(['adam', 'sgd']))
HP_L_RATE= hp.HParam('learning_rate', hp.Discrete([0.0005, 0.001]))

def train_model(hparams):
    model= tf.keras.models.Sequential([
    tf.keras.layers.InputLayer(input_shape=(None,43), dtype=tf.float64), 
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(hparams[HP_NUM_UNITS], return_sequences=True)),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(hparams[HP_NUM_UNITS], activation='relu')),   


    callback = tf.keras.callbacks.LearningRateScheduler(hparams[HP_L_RATE])

    model.fit(train_dataset, epochs=100,callbacks=[callback])  #, validation_data=test_dataset
    return loss

Problem is I cannot figure out how and where to insert hparams[HP_L_RATE] in the train_model().As you can see, I have tried to use a callback function to implement hparams[HP_L_RATE], but it does not work.

Thank you,


1 Answer 1


I guess the error in the example is that, instead of numbers objects were passed to HParams. You should handle the learning rate just as any other float parameter and just pass it to the optimizer. Actually hard to tell what you want exactly without a code example.

  • $\begingroup$ I have edited my question,added some codes! $\endgroup$
    – Arwen
    May 30, 2020 at 9:55
  • $\begingroup$ What do you exactly mean with 'You should handle the learning rate just as any other float parameter and just pass it to the optimizer'? $\endgroup$
    – Arwen
    May 31, 2020 at 7:36
  • 1
    $\begingroup$ you should add for example keras.optimizers.Adam(learning_rate=hparams[HP_L_RATE]) as a parameter in model.compile. The keras callback doesn't make sense since it should be based on an epoch $\endgroup$ May 31, 2020 at 9:09
  • $\begingroup$ The problem is I cannot use an optimizer name because the optimizer itself should be defined with 'hparams[HP_OPTIMIZER]'. Please check the line of code: 'model.compile (optimizer=hparams[HP_OPTIMIZER] ,loss='mae', metrics=['accuracy'])'. Therefore I am not able to pass learning rate with hparams inside the optimizer func. $\endgroup$
    – Arwen
    Jun 2, 2020 at 7:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.