# Why do we use the RELU activation function?

They prove that by using arbitrary bounded and non-constant functions then, we can approximate anything function we want which is the heart of machine learning. So I wonder why RELU is used because it would directly violate the theorem as RELU is unbounded.

I understand the reasoning that people use for using RELU that it combats exploding gradients of other activations and seems to work well etc. I am mainly confused about the activation functions that are not bounded as it violates the theorem, maybe I haven't delved deep enough into the literature to learn this

• Does this answer your question? Why ReLU is better than the other activation functions Jan 31 at 17:59
• @timmy1691, the theorem does not say that unbounded activations cannot model arbitrary functions. It just focuses on some conditions and derives some conclusions. However, proving A → B is not equivalent to proving !A → !B.
– noe
Feb 1 at 6:54
• @noe yes i realized that a few hours after posting the question, i wonder if there is a proof that proves the ¬A to ¬B? Feb 1 at 8:30
• Not that I am aware of. But this article offers a proof of unbounded → universal approximation. I also find this answer relevant.
– noe
Feb 1 at 8:58