4
$\begingroup$

I read about the Rosenblatt Perceptron Learning Algorithm. Often there is an explicit note:

It is important to note that all weights in the weight vector are being updated simultaneously

But why are all weights updated simultaneously? I tried another approach where I iterated over all weights and updated them in different iterations. It also worked on some simple test cases.

Could someone explain to me, why they are updated simultaneously and why should this approach be better?

$\endgroup$
3
$\begingroup$

The algorithm works by adding or subtracting the feature vector to/from the weight vector. If you only add/subtract parts of the feature vector your a not guaranteed to always nudge the weights in the right direction, which could mess with the convergence of the procedure.

The idea is that in the weight space every input vector is a hyperplane. You need to find a weight vector that is on the correct side of all the hyperplanes of your data inputs. The correct weight therefore is in a convex cone. If you observe a misclassification, that means your weight vector is on the wrong side of the hyperplane and therefore outside the convex cone of possible solutions.

Example of two example input (blue lines) with perpendicular vectors indicating on which side the output is correct

Now, by adding/subtracting the input vector to the weight vector you make sure that this data input vector is now correctly classified. You also make sure, that you decrease the distance of your weight vector by a sufficient margin (at least the length of the input vector) towards the cone of all possible solutions.

Example of a weight update

If you only add parts of the vector (i.e. not update all weights) at each iteration, you cannot be sure that you make sufficient progress or even move in the right direction of the solution space.

| improve this answer | |
$\endgroup$
  • $\begingroup$ thanks for the answer, is there any mathematical description for this? $\endgroup$ – Kevin Wallis Aug 4 '17 at 6:07
  • $\begingroup$ here is proof of convergence (pp.15) where they call the margin $\delta$ $\endgroup$ – oW_ Aug 4 '17 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.