Why is scale parameter on batch normalization not needed on ReLU?

Question

I now read a book titled "Hands-On Machine Learning with Scikit-Learn and TensorFlow" and on the chapter 11, the author writes the following explanation on batch normalization:

Note that by default batch_norm() only centers, normalizes, and shifts the inputs; it does not scale them (i.e., γ is fixed to 1). This makes sense for layers with no activation function or with the ReLU activation function, since the next layer’s weights can take care of scaling, but for any other activation function, you should add "scale": True to bn_params.

The batch_norm function is from TensorFlow, FYI. The author explains that the γ parameter should not be set on ReLU activation function.

However, I don't understand why on ReLU, the next layer's weights can take care of scaling... I understand that the next layer takes input from the ReLU output, which is max(0, x). Why does it take care of scaling and thus no need to set γ parameter on ReLU?

Answer 1

Scaling in batch normalization is said not to be required because of the nature of ReLU. Although it is a non-linear operation, a multiplication with a positive scalar $\gamma$ over the values of a vector $x$ which are already normalized and centered to the trained bias $\beta$ makes no influence on its non-linear outcome: for $\gamma > 0$, $max(0, \gamma \times x) = \gamma \times max(0, x)$. In fact, the extra parameter adds no representative power in the model, for the same reason that we must include non-linearities between other neuron layers.

And so, the weights of the following layer would scale the values to fit accordingly. We could have $z = Wgx$, but that would be the same thing as $z = Wx$ if $W$ is scaled by $g$. The same reasoning can't be made for other non-linearities, such as the sigmoid or tanh, which have a saturating effect that is clearly influenced by a scalar.

Curiously, the official documentation on TensorFlow's tf.contrib.layers.batch_norm makes the same recommendation:

• scale: If True, multiply by gamma. If False, gamma is not used. When the next layer is linear (also e.g. nn.relu), this can be disabled since the scaling can be done by the next layer.