While learning batch normalization, I was thinking why can't we solve the "gradient scale problem" by using an appropriate activation function ?

Like can't we delay and scale the activation function instead of scaling the whole dataset and ensure that the variance is preserve through it ?

  • 1
    $\begingroup$ I guess you are a bit wrong, you are talking about data normalization rather than batch normalization, the former is a pre-processing step. $\endgroup$ – Media Mar 3 '18 at 9:24
  • $\begingroup$ I was talking about normalising at each layer "automatically" or normalizing at the beginning and keep data's shape through layers. $\endgroup$ – Totem Mar 4 '18 at 7:01

What you describe sounds a lot like Scaled-Exponential Linear Units (SELUs), that are the core of Self-Normalizing Neural Networks, which where presented at NIPS 2017.

A short summary from here is that:

If the mean and variance of the input is in certain range, then the mean and variance of the output should (1) also in that range and (2) converge to a fixed point after iteratively applying the activation function.

You might want to have a look at the reddit post comments. If you want to fully understand them, you can go ahead with the 90 page-long appendix of the arxiv preprint.

They got a lot of attention when they were presented, but I think they have not delivered up to the expectations, as no one seems to be talking about them lately on the internet.

  • $\begingroup$ It was published six months after the edition of my textbook. The importance to follow the news ! Thanks for the details. Is there a reason why "no one seems to be talking about them" ? $\endgroup$ – Totem Mar 2 '18 at 21:12
  • $\begingroup$ I don't know whether there are technical reasons or not, but maybe people is getting skeptical due to the "grad student descent" practices lately, which makes it more difficult to get adoption unless spectacular and consistent SOTA results. $\endgroup$ – ncasas Mar 2 '18 at 21:20
  • $\begingroup$ My experience with elu confirms your last statement: its performance is very much like relu, not much better or worse, but slower. $\endgroup$ – Maxim Mar 5 '18 at 15:41
  • $\begingroup$ I missed your answer @ncasas, thanks I'll check that out. $\endgroup$ – Totem Mar 15 '18 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.