Skip to main content
edited tags
Link
Neil Slater
  • 29.2k
  • 5
  • 81
  • 100
Source Link
Blaszard
  • 911
  • 1
  • 13
  • 30

Why is scale parameter on batch normalization not needed on ReLU?

I now read a book titled "Hands-On Machine Learning with Scikit-Learn and TensorFlow" and on the chapter 11, the author writes the following explanation on batch normalization:

Note that by default batch_norm() only centers, normalizes, and shifts the inputs; it does not scale them (i.e., γ is fixed to 1). This makes sense for layers with no activation function or with the ReLU activation function, since the next layer’s weights can take care of scaling, but for any other activation function, you should add "scale": True to bn_params.

The batch_norm function is from TensorFlow, FYI. The author explains that the γ parameter should not be set on ReLU activation function.

However, I don't understand why on ReLU, the next layer's weights can take care of scaling... I understand that the next layer takes input from the ReLU output, which is max(0, x). Why does it take care of scaling and thus no need to set γ parameter on ReLU?