6
$\begingroup$

Today I've seen many Perceptron implementations with learning rates. According to Wikipedia:

there is no need for a learning rate in the perceptron algorithm. This is because multiplying the update by any constant simply rescales the weights but never changes the sign of the prediction

Is there some benefit to implementing a learning rate with Perceptron? If there is not, why do so many implementations have it?

$\endgroup$
4
$\begingroup$

I agree with Dawny33, choosing learning rate only scales w.

While training of Perceptron we are trying to determine minima and choosing of learning rate helps us determine how fast we can reach that minima. If we choose larger value of learning rate then we might overshoot that minima and smaller values of learning rate might take long time for convergence.

It is okay in case of Perceptron to neglect learning rate because Perceptron algorithm guarantees to find a solution (if one exists) in an upperbound number of steps, in other implementations it is not the case so learning rate becomes a necessity in them.

It might be useful in Perceptron algorithm to have learning rate but it's not a necessity.

$\endgroup$
2
$\begingroup$

The choice of learning rate m does not matter because it just changes the scaling of w.

I agree that it is just the scaling of w which is done by the learning rate.

Having said that, as I have explained in this answer, the magnitude of learning rate does play a part in the accuracy of the perceptron.

$\endgroup$
2
$\begingroup$

With regard to the single-layered perceptron (e.g. as described in wikipedia), for every initial weights vector $w_0$ and training rate $\eta>0$, you could instead choose $w_0'=\frac{w_0}{\eta}$ and $\eta'=1$.

For the same training set, training a perceptron with $w_0,\eta$ would be identical to training with $w_0',\eta'$, in the sense that:

  • Both perceptrons would make exactly the same mistakes.
  • After every mistake, each perceptron would update $w$ such that it would define the same hyperplane as the other perceptron.
  • Both perceptrons would make the same amount of mistakes until convergence.

(For a partial proof and code example, see here.)


Thus, in case $w_0=0$, the learning rate doesn't matter at all, and in case $w_0\not=0$, the learning rate also doesn't matter, except that it determines where the perceptron starts looking for an appropriate $w$.

So although tuning the learning rate might help to speed up the convergence in many other learning algorithms, it doesn't help in the case of the simple version of single-layered perceptron.

$\endgroup$
1
$\begingroup$

To clarify (for people like myself who are learning from scratch and need basic explanations), what Wikipedia means (if you look through the source) is that the learning rate does not affect eventual convergence, assuming the learning rate is between 0 and 1. A learning rate too large (example: consider an infinite learning rate where the weight vector immediately becomes the training case) can fail to converge to a solution.

The learning rate can, however, affect the speed at which you reach convergence (as mentioned in the other answers).

$\endgroup$
  • $\begingroup$ Please provide a source about how the perceptron can fail to converge if the learning rate is too large. I personally know that a positive learning rate is sufficient for it to converge. $\endgroup$ – MattSt Mar 9 '18 at 17:29
  • $\begingroup$ Matt, one source off the top of my head is the Google Developer Machine Learning Crash Course. They have a nice sandbox set of exercises that let you visualize the impact of the learning rate; I found it very helpful in understanding. $\endgroup$ – Silent Zebra Sep 20 '18 at 23:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.