You might want to check out this article on Neural Networks, Manifolds, and Topology for a "geometric interpretation of MLP output".
Here is an excerpt that you might be most interested in:
There are a variety of different kinds of layers used in neural networks. We will talk about tanh layers for a concrete example. A tanh layer tanh(Wx+b) tanh(Wx+b) consists of:
1.A linear transformation by the “weight” matrix W
2.A translation by the vector b
3.Point-wise application of tanh.
and here is an animated gif of the geometric description he gives:
At the beginning of this animation, you will see the lines are straight, and undergo a linear transformation (weights are multiplied by inputs, Wx) where the lines move, but remain parallel, with the origin also remaining at (0,0); this is what happens in all linear transformations.
Then, you will see the transformation translated by the bias vector b. That is, you will see all the lines move together in the direction of the bias vector b. In this case, you will see the entire picture move down and to the right.
Finally, you will see the "kernel trick" which transforms the inputs to the next layer in a nonlinear way, causing the nice parallel lines to warp. This non-linear warping allows the neural network to solve non-linear problems. That is, if the space is distorted, and the points you are trying to classify move apart under this non-linear transformation, it may then be possible to create a separating hyperplane between points which should have different classifications, thus allowing the neural network to differentiate between your classes.
Hopefully, this will give you a geometric interpretation of what is happening at each layer of a neural network. From there, you can extrapolate as you add more and more layers to your network.
In the example shown above, he is using tanh for the "kernel trick", but a number of other nonlinear functions could also be used (e.g. Sigmoid, ReLU, etc.)
If you find this animated gif hard to follow, I would strongly recommend you check out 3Blue1Brown's Essence of linear algebra series where he has a number of videos with intuitive explanations of Linear Algebra and great geometric animations.
