Why is activation needed at all in neural network?

Question

I watched the Risto Siilasmaa video on Machine Learning. It's very well explained, but the question emerged that at what stage should we use the activation function and why we need it at all. I know that by definition the activation function transforms the sum of w*x+b to a number between some lower and upper limit.

In the video Risto Siilasmaa explains that in the training process there are the following steps:

1. Start with random weights.
2. Calculate the outcome (sum of w*x+b) - we know what it should be because we know what image we gave to the system.
3. Calculate the error.
4. Nudge all weights to reduce the error.

But what about the activation? Where to place it in the previous list? Before the error calculation? And what would happen if we omitted it altogether? Just calculate the outcome and error and nudge the weights? Is it because the error calculation doesn't work well when the value of outcome isn't transformed between some lower and upper limit?

Answer 1

Generally the activation is part of the model and gets applied for each neuron, so definitely before the error calculation. What the activation function is depends on what task you are solving and where the neuron of interest is. In principle the activation function $f$ would go to the calculation of the outcome

$$ y = f(Wx + b)$$

For output neurons, if you are doing classification, then $f$ should map between 0 and 1, since you'll interpret the outcome as a probability. For regression $f$ could be just the identity.

For hidden (i.e. non-output neurons), you definitely want to use a non-linear $f$. The reason is that the neural network would otherwise be equivalent to a regular linear model. So the non-linear activations are needed to harvest the expressive power of neural networks.

For deep learning the most popular $f$ for hidden neurons would probably be the rectified linear unit (relu)

$$ f(x) = \max(0,x)$$

Answer 2

Activation function is applied after (sum of w*x+b) for each neuron in each layer.

The role of activation function is to introduce non linearity "higher order relation ship" between inputs and outputs.

Answer 3

Without activation, the model is just a linear model like linear plotting,regression. Where is the "learning"?

2.Calculate the outcome (sum of w*x+b) - we know what it should be because we know what image we gave to the system.

The weights are random. The neuron does not know how to bound the value (the firing pattern). Activation should be there as an instruction how to bound the output. Otherwise from layer to layer outcome can be anything.

4. Nudge all weights to reduce the error.

How we do it? We find out the gradient (direction of minimizing/maximizing) because if you want to optimize the function as desired , you find its values at its derivative/gradient, back propagate the gradient since we want to minimize or maximize some cost function (error or difference in this case). How will you find gradient of a linear function since differentiating it will be a constant. Learning will essentially stop.

So we introduce certain non-linearity (activation functions) to the system so that the gradient varies and we get a way to update our weights every time we back propagate.

Answer 4

Mathematically, the weights that sit between two given rows of neurons collectively form a transformation matrix, and a row of neurons forms a vector. To use the network, we use the matrix to transform the vector, giving us a vector representing the next row of neurons. Then we apply an activation function to those neurons. Then we proceed to the next layer and repeat.

So what happens when we don't have an activation function? Then we just have a series of matrix transformations, and we can use matrix multiplication to compute a single matrix that does the same thing. So in truth such a network has no hidden layers, and is incapable of deep learning.

Answer 5

Talking about activation...
As we know that the activation function gives the positive signal only then when the input (sum of weights and inputs) it gets is positive value. Am I right? Consequently the weights we use to multiply pixel input values has to be as big as possible. But why is it this way that only positive value fits to give better probability value? Why smaller weights are not suitable?

Answer 6

Ok, let's put it like that: "no activation" means "linear model", i.e. a model that can only learn linear associations between your variables. This is a very big limitation! The whole point of Neural Networks is: almost any regularity we observe in the world is non-linear, therefore we must make them flexible enough to learn all these complex patterns. That's where non-linear transformations come into play.

A bit more technically, a layer is like a function:

output = f(Weights * Data + Bias)

if all your f()'s in your layers are linear, even the output of the stack of layers such as:

ANN_output = f_3( f_2( f_1(Weights * Data + Bias) ) ) # yeah that's a Neural Net

will be linear as well.

Non-linear activation functions let us overcome this limitation, and allow a model to learn patterns that could be either linear or not. And the more you stack layers (i.e. the more you nest functions into one another) the more complex the final non-linear transformation becomes, making the model able to learn more complicated things.