1
$\begingroup$

Using full-batch gradient descent, stacking 100 layers and using alpha 0.0001 results in steadily decreasing error.

However, after I implemented Batch Norm, the same scenario results in fluctuations. My implementation was verified by several people, so now I am wondering - why batch norm adds this stochastisity effect?

In fact, I am no longer able to stack 100+ layers, only approximatelly 10 layer before stochasticity becomes very apparent and hard to control.

What's more interesting -it seems to get worse with smaller learning rate! 0.4 is good, and 0.0001 results in smaller updates (as expected) but larger relative fluctuation (caught me by surprise).

Why is it so?

Edit: Just tried 100 layers (each 63 neurons). It's very noisy, but I am able to more-or-less steadily reduce the error. It's less stochastic if I set learning rate to 10 ("ten", lol!) and will get only noisier if it's say 0.5 which is incredibly weird

Of course, this learning rate is so high that the error does occasionally snap to a high value, but seems to work with 100 layers...

Notice - I am using full-batch gradient descent, with 50 elements

Edit:

Using 100 LSTM units, stacked like a pillar onto each other.

Number of timesteps is 50. Feature size 50 (lstm state has dimension of 50), because my sentence has 50 distinct characters, each character is encountered only once per epoch.

Performing backpropagation after 50 timesteps Using vanilla gradient descent, all fancy things like accelerated momentum or L2 norm, dropout are disabled.

I am sure there is no error, but interested if anyone can tell why Batch Norm has this stochastic property, if that's common with people.

Improve this question
$\endgroup$
4
  • $\begingroup$ What do you mean with "full-batch" gradient descent? From your last sentence "Notice - I am using full-batch gradient descent, with 50 elements" I would say you are doing mini-batch gradient descent... $\endgroup$
    – noe
    Mar 15, 2018 at 16:44
  • $\begingroup$ I have 50 data elements total, so all fits into a single batch $\endgroup$
    – Kari
    Mar 15, 2018 at 17:15
  • $\begingroup$ When you say you implemented Batch Norm, is that your own code, or something from a library e.g. Keras? I ask because although I expect some additional variation due to Batch Norm, some of your results might be explained by a problem with the implementation $\endgroup$
    – Neil Slater
    Mar 15, 2018 at 20:46
  • $\begingroup$ My implementation, but it's highly unlikely. Without batch norm there are no fluctuations at all. And the batch norm code itself was validated by 2 more people. What might cause this additional variation? $\endgroup$
    – Kari
    Mar 16, 2018 at 5:20

3 Answers 3

Reset to default
2
$\begingroup$

While I am no expert in batch normalization, I have noticed that in the paper they include a constant, epsilon (ε) to the minibatch variance for numerical stability (in the normalize step from this paper, page 3). So while your math might be right and verified by others, the actual computation could still be wrong.

This issue appears here. The solution on the bottom is to add variance clipping.

And with a stab at why you might be seeing it more with smaller learning rates. Well that might be just because you're taking a lot more steps to get to the same accuracy so that's just many more steps of numerical instability. EDIT: I now realize when you say full-batch gradient descent you mean you are doing the entire dataset before updating the weights (just standard gradient descent). This means that with a smaller learning rate you will have to be doing more steps... so I'm sure that explains the learning rate difference.

Improve this answer
$\endgroup$
1
$\begingroup$

What about trying to use a BN implementation provided by a widely use lib e.g. Keras instead of yours, to test your implementation correctness? If after this change you get different observations then it's possible your implementation is in some way different to the standard one

About the learning rate, one of the advantages of BN is that it should allow to use higher learning rate because it should combat possible scale explosion happening because of this

Having more information about the NN architecture could help

Improve this answer
$\endgroup$
0
0
$\begingroup$

I don't necessarily think it's that batch normalization is necessarily stochastic, but rather just how batch normalization works when compared to the inference stage. As you may know, during the training phase batch normalization depends on the mini-batch. However, that dependency is undesirable for inference so instead a moving average over all mini-batches is taken.

Doing this obviously can cause problems since what your model is inference on is different than what it was trained on. Furthermore, the smaller your mini-batch the worse the performance gets with more depth since the the inaccuracies computed for the smaller batch just get compounded more and more.

If you insist on having a batch normalization phase, a way to help mitigate this might include implementing a batch re-normalization setup instead.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks, in my post I am having issues during training. I am using full batch gradient descent, not minibatch. Due to this I also don't have a running/moving average, because each epoch encounters same element at the same place $\endgroup$
    – Kari
    Mar 19, 2018 at 14:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.