I watched the Risto Siilasmaa video on Machine Learning. It's very well explained, but the question emerged that at what stage should we use the activation function and why we need it at all. I know that by definition the activation function transforms the sum of w*x+b to a number between some lower and upper limit.
In the video Risto Siilasmaa explains that in the training process there are the following steps:
- Start with random weights.
- Calculate the outcome (sum of w*x+b) - we know what it should be because we know what image we gave to the system.
- Calculate the error.
- Nudge all weights to reduce the error.
But what about the activation? Where to place it in the previous list? Before the error calculation? And what would happen if we omitted it altogether? Just calculate the outcome and error and nudge the weights? Is it because the error calculation doesn't work well when the value of outcome isn't transformed between some lower and upper limit?