1
$\begingroup$

I don't understand the following statement:

The choice of learning rate m does not matter (for Perceptron) because it just changes the scaling of w (weights). The site with this statement that wikipedia cites.

The update rule of perceptron is $w \pm x$ for mistaken cases. If the update to the weight vector is addition or subtraction, how can it only scale the weights?

$\endgroup$
5
  • $\begingroup$ Please edit your question and move the answer in a separate answer. $\endgroup$ – Marmite Bomber Sep 10 '18 at 23:41
  • $\begingroup$ A perceptron does not need to use hinge loss, you can use any loss function you see fit. $\endgroup$ – JahKnows Sep 11 '18 at 2:10
  • $\begingroup$ Do you mean that any loss function giving good gradient can be used? $\endgroup$ – user2573741 Sep 11 '18 at 20:15
  • $\begingroup$ @user2573741, precisely. Many different loss functions can be used with the perceptron. Your question depends on the loss function used. Perceptron is a single neuron neural network, a very common loss function to use is the root mean squared error, in which case the learning rate very much affects the outcome of your graident descent optimization. $\endgroup$ – JahKnows Sep 11 '18 at 22:28
  • $\begingroup$ I'm not sure one can use mean squared error as the cost function for Perception. The reason is that the decision function (or activation) of Perceptron is a step function, and it's derivative is infinite at 0. In other words, the gradient of mse for Perceptron is bad. MSE for perceptron is very close to the 0/1 loss of Perceptron, but again it has no good gradient. $\endgroup$ – user2573741 Sep 11 '18 at 23:31
3
$\begingroup$

After some study, I figured out the answer and want to share with people if someone also finds it helpful. The loss function of Perceptron is hinge loss or

$J(w) = max(0, -yw^Tx)$.

Adding a constant to the loss function does not change the function value, as it does not change the sign of the decision. In other words

$J_2(w) = max(0, -\alpha yw^Tx) = J(w)$.

If we do gradient descent using $J_2$, we have

$\partial(J_2)/\partial(w) = 0$, if $J_2 = 0$;

$\partial(J_2)/\partial(w) = -\alpha yx$, otherwise.

So the update function of gradient descent is

$w_{new} = w_{old} \pm \alpha x$.

As long as $\alpha > 0$, it does not change Perceptron decision in any step. This is why for Perceptron, you only need to set learning rate to be 1.

Specifically answer the question, when people say "the learning rate only scales $w$", they are referring to $J_2(w) = max(0, -\alpha yw^Tx)$ rather than $w_{new} = w_{old} \pm \alpha x$.

A related question I found very helpful is Normalizing the final weights vector in the upper bound on the Perceptron's convergence

$\endgroup$
0
$\begingroup$

The learning rate regulates the amount by which the weights will change at every step $t$ of the gradient descent algorithm.

It is not true that it can be set to any arbitrary amount, this is furthest from the truth. A learning rate which is too small will never allow your machine learning model to converge to a minimum, and a learning rate which is too large will cause your model parameters to oscillate around a possible minimum.

I answered in more details here how the basic gradient descent algorithm is affected by the choice of the learning rate and then i propose some better alternative methods: Do adaptive learning optimizers follow the steepest decent?.

More details

More accurately the update rule for gradient descent is

$w^{t+1} = w^t + m\nabla J(w)$

where $m$ is the learning rate and

$J(w) = \frac{1}{2}\sum(y-\hat{y})^2$

is the cost function. Thus, it does not scale the weights in a multiplicative sense, however it does scale the correction the weights will experience at each step of the algorithm.

$\endgroup$
3
  • 1
    $\begingroup$ This does not answer the question. I understand gradient descent and the role of learning rate. I'm asking why the learning rate value does not matter for Perceptron. $\endgroup$ – user2573741 Sep 6 '18 at 16:47
  • 1
    $\begingroup$ I answered you that it in fact DOES matter. It matters very much. Read the second paragraph again. "It is not true that it can be set to any arbitrary amount, this is furthest from the truth. A learning rate which is too small will never allow your machine learning model to converge to a minimum, and a learning rate which is too large will cause your model parameters to oscillate around a possible minimum." $\endgroup$ – JahKnows Sep 11 '18 at 2:09
  • 1
    $\begingroup$ For perceptron, the actual value of learning rate does NOT make difference (as long as it is positive). Every update step in Perceptron learning is taken when a prediction mistake happens, and the algorithm converges when there is no more mistake. Since the prediction correctness is irrelevant to learning rate, the learning rate will not impact training time. In fact, learning rate is not in the formula of Perceptron convergence upper bound. So even it is set to a very small number, slow training will not happen as a result. $\endgroup$ – user2573741 Sep 11 '18 at 19:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.