Skip to main content
1 of 3
noe
  • 27.5k
  • 1
  • 49
  • 80

The point of using activations is to enable the network to learn non-linear functions. If you do not use activations and just stack linear layers, the result is equivalent to a single linear layer. This is the mathematical proof with 2 linear layers (extendable to N by induction):

$ y = (xW_1 + b_1) W_2 + b_2 = x W_1 W_2 + (b_1 W_2 + b_2) = x W' + b'$

(where $W'= W_1 W_2$ and $b'= b_1 W_2 + b_2$)

noe
  • 27.5k
  • 1
  • 49
  • 80