I'm writing up a neural network using sigmoid as the activation function. According to one lecture, sigmoid simply squashes numbers onto the (0,1) interval. To model non-linear decision boundaries, you need high dimensional data. Another lecture, this one on sigmoid in neural nets context (MIT S.191), suggested sigmoid's non-linear behaviour is the reason why a neural net can model a non-linear decision boundary. So, I'm a bit confused. Does sigmoid facilitate modeling non-linear decision boundaries or does this come from high-dimensional data because you can use the sigmoid to produce a linear decision boundary w/o incident?
1 Answer
$\begingroup$
$\endgroup$
The power to model non-linear decision boundaries comes directly from the non-linear activation function. You can understand this when you see that concatenating N linear transformations (i.e. dense layers) is equivalent to a single linear transformation. This is the mathematical proof with 2 linear layers:
$ y = (xW_1 + b_1) W_2 + b_2 = x W_1 W_2 + (b_1 W_2 + b_2) = x W' + b'$
(where $W'= W_1 W_2$ and $b'= b_1 W_2 + b_2$)
