I am using TensorFlow to implement some basic ML code in Python. I was wondering if anyone could give me a short explanation of the meaning of and difference between step size and learning rate in the following functions.

I used tf.train.GradientDescentOptimizer() to set the parameter learning rate and linear_regressor.train() to set the number of steps. I've been looking through the documentation on tensorflow.org for these functions but I still do not have a complete grasp of the meaning of these parameters.

Thank you and let me know if there is any more info I can provide.

(I posted this on Stack Overflow before I knew there was a Data Science forum board too, sorry)


Both learning rate $\eta$ and step size $\Delta w$ are linked to gradient descent. In the most simple case they are linked by :

$$ \Delta w = w(t+1)-w(t) = -\eta \frac{\partial E(w)}{\partial w}$$

Where $t$ is the epoch and $E$ the error function.

In that simple case, they only differ by $ - \frac{\partial E(w)}{\partial w} $, which sometimes lead to use one term instead of the other. However in a more general case (learning rate depending on weights, learning rate depending on epoch, added momentum, or minibatch learning) the distinction may have a greater importance.

Edit : I answered the title of your question, in the body you seems to refer to the number of step, which is the number of iteration or number of epoch of the aforementionned formula.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.