5
$\begingroup$

Batch normalization is generally preferred in deep learning, which normalizes the output of the activation function in each layer (as an output from the cost function differs depends on the input).

Instead, if the training set is normalized before passing through layers, does this solve the same problem?

$\endgroup$
5
$\begingroup$

Batch normalization supposedly solves the issue of covariate shift within each layer. Even if you start out with a normalized/whitened dataset, after an input is passed through a layer of the network it may no longer be centered around 0 with unit variance.

For example, if one considers a layers of the same size with constant weight matrix $w_{ij}=10$ and bias $b_j = 4$, so that $z^{L}_i = \sum_{j}w_{ij}z^{L-1}_j + b^L_j$, then if the neurons in layer $L-1$ (i.e. $z^{L-1}_i$) have zero mean and unit variance, then the neurons in layer $L$ will have variance of 10 and mean of 4 (here I'm ignoring the activation function for simplicity).

Therefore even if the input is normalized, the distribution of neuron values within the network are not guaranteed to be normalized. Batch norm solves this issue by normalizing each layer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.