0
$\begingroup$

I'm writing up a neural network using sigmoid as the activation function. According to one lecture, sigmoid simply squashes numbers onto the (0,1) interval. To model non-linear decision boundaries, you need high dimensional data. Another lecture, this one on sigmoid in neural nets context (MIT S.191), suggested sigmoid's non-linear behaviour is the reason why a neural net can model a non-linear decision boundary. So, I'm a bit confused. Does sigmoid facilitate modeling non-linear decision boundaries or does this come from high-dimensional data because you can use the sigmoid to produce a linear decision boundary w/o incident?

$\endgroup$
1
$\begingroup$

The power to model non-linear decision boundaries comes directly from the non-linear activation function. You can understand this when you see that concatenating N linear transformations (i.e. dense layers) is equivalent to a single linear transformation. This is the mathematical proof with 2 linear layers:

$ y = (xW_1 + b_1) W_2 + b_2 = x W_1 W_2 + (b_1 W_2 + b_2) = x W' + b'$

(where $W'= W_1 W_2$ and $b'= b_1 W_2 + b_2$)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.