7
$\begingroup$

I am a trying to understand the SELU activation function and I was wondering why deep learning practitioners keep using RELU, with all its issues, instead of SELU, which enables a neural network to converge faster and internally normalizes each layer?

$\endgroup$
2
  • $\begingroup$ This may help you. $\endgroup$ Oct 2 at 19:33
  • $\begingroup$ Took me a while to answer. Undeniably, the "dead relu" can be an asset for multiple reasons. But if there are enough 0's during the training process, we can have a neural network that does not learn anything. Another problem that is not avoided by ReLU, is the issue of exploding gradients, which is impossible for SELU. $\endgroup$ Oct 5 at 8:35
3
$\begingroup$

ReLU is quick to compute, and also easy to understand and explain. But I think people mainly use ReLU because everyone else does. The activation function doesn't make that much of a difference, and proving or disproving that requires adding yet another dimension of hyperparameter combinations to try.

If the research is for a paper there is another consideration: you will want to stick with what your benchmarks use, what everyone else is doing, unless the research is specifically about activation functions.

(As an aside, I see practically no research on the pros or cons of using different activation functions at different layers. I suspect this is also because of the hyperparameter combinatorial explosion, combined with the expectation of it not making much difference.)

The SELU function is a hard-sell in a couple of ways. First it requires reading a long paper to understand, and accept the couple of magic numbers it comes with. But a bigger factor might be that it does internal normalization, meaning you don't need your batch or layer normalization any more. Or do you? Suddenly this is not a simple swap in for ReLU, but affects other parts of the architecture.

This is a good article on a large selection of alternative activation functions: https://mlfromscratch.com/activation-functions-explained/ The con they give there for SELU is that there are not enough comparative research papers on it, for different architectures, yet.

$\endgroup$
3
  • $\begingroup$ I think it's the same as sigmoid or tanh. Everyone were using them but then some papers started using ReLU so everyone turned their attention there. I am not saying this is a similar case, but based on your answer and what I have concluded myself, researchers shouldn't be afraid to explore an uncharted area. As for the practical side of your answer, I agree that SELU is hard to understand, hence the question, but I disagree on the fact that activation functions will not make any difference because of the hyperparameter combination. $\endgroup$ Oct 6 at 9:23
  • $\begingroup$ @KonstantinosSkoularikis I was trying to answer the question you asked, which was why humans choose ReLU. I get the sense the question you really wanted to ask is more along the lines of: for what kind of models or architectures or domains is ReLU provably superior to SELU? You could post that as a new question, it fits the StackOverflow approach. (It might remain unanswered until someone has done the research, though.) $\endgroup$ Oct 8 at 8:09
  • $\begingroup$ In this context, you are right @Darren Cook. I will mark your question as correct. Thank you for the nice resource on activation functions. $\endgroup$ Oct 8 at 10:26
0
$\begingroup$

I want to answer this question with respect to my experience with scientific papers. The point is that when practitioners try to make new ideas, they should have ablation study in their work. This means that they should satisfy the readers and the reviewers that the claimed improvements are real. They should concentrate on the novelty of their work. This means that in papers, and probably their implementations, you usually do not see state-of-the-art modules. They leverage trivial modules and try to show the effectiveness of their own new-designed or manipulated module(s). This is why everyone knows there are so many optimisation approaches that are better than Adam but there are still numerous novel papers that exploit Adam for optimisation.

On the other hand, if a scientist uses a typical state-of-the-art approach alongside his novel approach, he should prove that the improvement is not solely due to the state-of-the-art things he's used that are apart from his original work. This means the ablation study should be longer. Students usually avoid this.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.