How do two perceptrons produce different linear decision boundaries?

Question

I'm trying to visualize how two perceptrons converge to two different decision boundaries (which is ultimately used to create the classifier for the non-linearly separable data).

Source: https://tdb-alcorn.github.io/2017/12/17/seeing-like-a-perceptron.html

I don't understand how the second perceptron creates a different decision boundary when it has the same input as the first perceptron? I know the weights can be initialized differently but does this second perceptron classify something else? Shouldn't the decision boundaries be converge to be the the same ultimately?

Source: https://tdb-alcorn.github.io/2017/12/17/seeing-like-a-perceptron.html

Answer 1

In short, because you are not training the individual hidden neurons, but the entire network.

Thinking holistically, your little net there is just a function with 9 parameters (two weights and a bias at each non-input node). Ignoring the structure of that function for now, you're just applying gradient descent (or maybe some variant thereof) to minimize some error function on your training set. There's no reason for this to converge to a point where the parameters at $p_0, p_1$ are the same.

When you compute the gradient, it's "at" $p_2$, so you're asking how to move the two decision boundaries (well, also the scaling of the scores on one side of the boundary, and how the two boundaries are combined into the final one) in order to improve the overall output best. Since the model will generally do better with a more complicated overall decision boundary, it will move the two hidden boundaries in different directions; it will not try to make them the same!

See also What is the purpose of multiple neurons in a hidden layer?

Answer 2

In the article which you linked, it seems that none of the two perceptrons are trained on data, you just have two perceptrons with different weights w0 and w1. If they have different weights, then they will have two different decision boundaries. You can see that, looking at the code that plots the decision boundaries:

ax.plot(t0, decision_boundary(w0, t0), 'm', label='Perceptron #0 decision boundary')
ax.plot(t1, decision_boundary(w1, t1), 'g', label='Perceptron #1 decision boundary')

where decision boundaries is

def decision_boundary(weights, x0):
    return -1. * weights[0]/weights[1] * x0

Answer 3

You have two forces working here: initial conditions and training.

Regarding training: in the end, we don't really care which of the two hidden-layer perceptrons become that near-vertical line in your second figure and which becomes the near-horizontal line - as long as convergence does happen.

Regarding which of the two perceptrons become which of the two decision boundaries in question, that takes us to network initialization.

See these lecture slides from the University of Sterling directly addressing your question. From slide 13:

The gradient descent learning algorithm treats all the weights in the same way, so if we start them all off with the same values, all the hidden units will end up doing the same thing and the network will never learn properly.

addressing your question. Followed by

For that reason, we generally start off all the weights with small random values. Usually we take them from a flat distribution around zero [–wt, +wt], or from a Gaussian distribution around zero with standard deviation wt.

Proposing a solution which I'm pretty sure is very common practice for these kind of networks. More useful notes on initialization are given the same slide.

Generally, there's a lot of effort invested in deciding what to do at initialization. Keras has many options around initializing, for example.