0
$\begingroup$

I reading about activation functions in feedforward neural networks. ad a really old paper https://web.njit.edu/~usman/courses/cs677_spring21/hornik-nn-1991.pdf.

They prove that by using arbitrary bounded and non-constant functions then, we can approximate anything function we want which is the heart of machine learning. So I wonder why RELU is used because it would directly violate the theorem as RELU is unbounded.

I understand the reasoning that people use for using RELU that it combats exploding gradients of other activations and seems to work well etc. I am mainly confused about the activation functions that are not bounded as it violates the theorem, maybe I haven't delved deep enough into the literature to learn this

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ Does this answer your question? Why ReLU is better than the other activation functions $\endgroup$
    – m13op22
    Commented Jan 31 at 17:59
  • $\begingroup$ @timmy1691, the theorem does not say that unbounded activations cannot model arbitrary functions. It just focuses on some conditions and derives some conclusions. However, proving A → B is not equivalent to proving !A → !B. $\endgroup$
    – noe
    Commented Feb 1 at 6:54
  • $\begingroup$ @noe yes i realized that a few hours after posting the question, i wonder if there is a proof that proves the ¬A to ¬B? $\endgroup$
    – timmy1691
    Commented Feb 1 at 8:30
  • $\begingroup$ Not that I am aware of. But this article offers a proof of unbounded → universal approximation. I also find this answer relevant. $\endgroup$
    – noe
    Commented Feb 1 at 8:58

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.