Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I read about the Rosenblatt Perceptron Learning Algorithm. Often there is an explicit note:
It is important to note that all weights in the weight vector are being updated simultaneously
But why are all weights updated simultaneously? I tried another approach where I iterated over all weights and updated them in different iterations. It also worked on some simple test cases.
Could someone explain to me, why they are updated simultaneously and why should this approach be better?
The algorithm works by adding or subtracting the feature vector to/from the weight vector. If you only add/subtract parts of the feature vector your a not guaranteed to always nudge the weights in the right direction, which could mess with the convergence of the procedure.
The idea is that in the weight space every input vector is a hyperplane. You need to find a weight vector that is on the correct side of all the hyperplanes of your data inputs. The correct weight therefore is in a convex cone. If you observe a misclassification, that means your weight vector is on the wrong side of the hyperplane and therefore outside the convex cone of possible solutions.
Now, by adding/subtracting the input vector to the weight vector you make sure that this data input vector is now correctly classified. You also make sure, that you decrease the distance of your weight vector by a sufficient margin (at least the length of the input vector) towards the cone of all possible solutions.
If you only add parts of the vector (i.e. not update all weights) at each iteration, you cannot be sure that you make sufficient progress or even move in the right direction of the solution space.
