1
$\begingroup$

In Tensorflow documentation, it is shown how to tune several hyperparameters but not the learning rate.I have searched how to tune learning rate using HParams dashboard but could not find much. The only example is another question on github but it does not work.Can you please give me some suggestions on this?Should I use a callback function?Or provide different learning rates in hp_optimizer as in the question in github? Or something else?

Parts of my code is below:

HP_NUM_UNITS = hp.HParam('num_units', hp.Discrete([16, 32]))
HP_OPTIMIZER = hp.HParam('optimizer', hp.Discrete(['adam', 'sgd']))
HP_L_RATE= hp.HParam('learning_rate', hp.Discrete([0.0005, 0.001]))


def train_model(hparams):
    model= tf.keras.models.Sequential([
    tf.keras.layers.InputLayer(input_shape=(None,43), dtype=tf.float64), 
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(hparams[HP_NUM_UNITS], return_sequences=True)),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(hparams[HP_NUM_UNITS], activation='relu')),   
    tf.keras.layers.Dense(1)])

    model.compile(optimizer=hparams[HP_OPTIMIZER],loss='mae',metrics=['accuracy'])

    callback = tf.keras.callbacks.LearningRateScheduler(hparams[HP_L_RATE])

    model.fit(train_dataset, epochs=100,callbacks=[callback])  #, validation_data=test_dataset
    _,loss=model.evaluate(test_dataset)
    return loss

Problem is I cannot figure out how and where to insert hparams[HP_L_RATE] in the train_model().As you can see, I have tried to use a callback function to implement hparams[HP_L_RATE], but it does not work.

Thank you,

$\endgroup$
0
$\begingroup$

I guess the error in the example is that, instead of numbers objects were passed to HParams. You should handle the learning rate just as any other float parameter and just pass it to the optimizer. Actually hard to tell what you want exactly without a code example.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I have edited my question,added some codes! $\endgroup$ – Arwen May 30 at 9:55
  • $\begingroup$ What do you exactly mean with 'You should handle the learning rate just as any other float parameter and just pass it to the optimizer'? $\endgroup$ – Arwen May 31 at 7:36
  • 1
    $\begingroup$ you should add for example keras.optimizers.Adam(learning_rate=hparams[HP_L_RATE]) as a parameter in model.compile. The keras callback doesn't make sense since it should be based on an epoch $\endgroup$ – MichaelRazum May 31 at 9:09
  • $\begingroup$ The problem is I cannot use an optimizer name because the optimizer itself should be defined with 'hparams[HP_OPTIMIZER]'. Please check the line of code: 'model.compile (optimizer=hparams[HP_OPTIMIZER] ,loss='mae', metrics=['accuracy'])'. Therefore I am not able to pass learning rate with hparams inside the optimizer func. $\endgroup$ – Arwen Jun 2 at 7:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.