While learning batch normalization, I was thinking why can't we solve the "gradient scale problem" by using an appropriate activation function ?

Like can't we delay and scale the activation function instead of scaling the whole dataset and ensure that the variance is preserve through it ?

  • 1
    $\begingroup$ I guess you are a bit wrong, you are talking about data normalization rather than batch normalization, the former is a pre-processing step. $\endgroup$ Commented Mar 3, 2018 at 9:24
  • $\begingroup$ I was talking about normalising at each layer "automatically" or normalizing at the beginning and keep data's shape through layers. $\endgroup$
    – 0xmax
    Commented Mar 4, 2018 at 7:01

1 Answer 1


What you describe sounds a lot like Scaled-Exponential Linear Units (SELUs), that are the core of Self-Normalizing Neural Networks, which where presented at NIPS 2017.

A short summary from here is that:

If the mean and variance of the input is in certain range, then the mean and variance of the output should (1) also in that range and (2) converge to a fixed point after iteratively applying the activation function.

You might want to have a look at the reddit post comments. If you want to fully understand them, you can go ahead with the 90 page-long appendix of the arxiv preprint.

They got a lot of attention when they were presented, but I think they have not delivered up to the expectations, as no one seems to be talking about them lately on the internet.

  • $\begingroup$ It was published six months after the edition of my textbook. The importance to follow the news ! Thanks for the details. Is there a reason why "no one seems to be talking about them" ? $\endgroup$
    – 0xmax
    Commented Mar 2, 2018 at 21:12
  • $\begingroup$ I don't know whether there are technical reasons or not, but maybe people is getting skeptical due to the "grad student descent" practices lately, which makes it more difficult to get adoption unless spectacular and consistent SOTA results. $\endgroup$
    – noe
    Commented Mar 2, 2018 at 21:20
  • $\begingroup$ My experience with elu confirms your last statement: its performance is very much like relu, not much better or worse, but slower. $\endgroup$
    – Maxim
    Commented Mar 5, 2018 at 15:41
  • $\begingroup$ I missed your answer @ncasas, thanks I'll check that out. $\endgroup$
    – 0xmax
    Commented Mar 15, 2018 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.