Geometric interpretation of MLP output

Question

I am really interested in the geometric interpretation of perceptron outputs, mainly as a way to better understand what the network is really doing, but I can't seem to find much information on this topic.

I know a perceptron with no hidden layers can be seen as defining a hyperplane in between two classes, which works fine if they are linearly separable.

However, I find more difficult to understand what exactly does a multi layer perceptron do, and how it can classify an input that is initially not linearly separable.

I found this page http://www.borgelt.net/doc/mlpd/mlpd.html that says the hidden layers can be interpreted as making a coordinate transformation on the input. I would like to read more about this, but I didn't find much online, so I would really appreciate if you could elaborate on this or point me to some paper or book that covers this.

I am also puzzled by the visualisation that appears here http://tiny.cc/2l7o7y , a 3 neuron hidden layer manages to separate the input, which makes perfect sense since separating that input in a 3 dimensional space is trivial. However, adding another 2 neuron layer after that creates in them shapes that are obviously non-linear. My guess is that it represents the separation created in the original 2-dimensional space. Is that right? If not, what are we seeing here, exactly?

Answer 1

You might want to check out this article on Neural Networks, Manifolds, and Topology for a "geometric interpretation of MLP output".

Here is an excerpt that you might be most interested in:

There are a variety of different kinds of layers used in neural networks. We will talk about tanh layers for a concrete example. A tanh layer tanh(Wx+b) tanh⁡(Wx+b) consists of:

1.A linear transformation by the “weight” matrix W

2.A translation by the vector b

3.Point-wise application of tanh.

and here is an animated gif of the geometric description he gives:

At the beginning of this animation, you will see the lines are straight, and undergo a linear transformation (weights are multiplied by inputs, Wx) where the lines move, but remain parallel, with the origin also remaining at (0,0); this is what happens in all linear transformations.

Then, you will see the transformation translated by the bias vector b. That is, you will see all the lines move together in the direction of the bias vector b. In this case, you will see the entire picture move down and to the right.

Finally, you will see the "kernel trick" which transforms the inputs to the next layer in a nonlinear way, causing the nice parallel lines to warp. This non-linear warping allows the neural network to solve non-linear problems. That is, if the space is distorted, and the points you are trying to classify move apart under this non-linear transformation, it may then be possible to create a separating hyperplane between points which should have different classifications, thus allowing the neural network to differentiate between your classes.

Hopefully, this will give you a geometric interpretation of what is happening at each layer of a neural network. From there, you can extrapolate as you add more and more layers to your network.

In the example shown above, he is using tanh for the "kernel trick", but a number of other nonlinear functions could also be used (e.g. Sigmoid, ReLU, etc.)

If you find this animated gif hard to follow, I would strongly recommend you check out 3Blue1Brown's Essence of linear algebra series where he has a number of videos with intuitive explanations of Linear Algebra and great geometric animations.

Answer 2

If you restrict to the heavy-side function, the output is completely interpretable, see this explanation.