Using full-batch gradient descent, stacking 100 layers and using alpha 0.0001 results in steadily decreasing error.
However, after I implemented Batch Norm, the same scenario results in fluctuations. My implementation was verified by several people, so now I am wondering - why batch norm adds this stochastisity effect?
In fact, I am no longer able to stack 100+ layers, only approximatelly 10 layer before stochasticity becomes very apparent and hard to control.
What's more interesting -it seems to get worse with smaller learning rate! 0.4 is good, and 0.0001 results in smaller updates (as expected) but larger relative fluctuation (caught me by surprise).
Why is it so?
Edit: Just tried 100 layers (each 63 neurons). It's very noisy, but I am able to more-or-less steadily reduce the error. It's less stochastic if I set learning rate to 10 ("ten", lol!) and will get only noisier if it's say 0.5 which is incredibly weird
Of course, this learning rate is so high that the error does occasionally snap to a high value, but seems to work with 100 layers...
Notice - I am using full-batch gradient descent, with 50 elements
Using 100 LSTM units, stacked like a pillar onto each other.
Number of timesteps is 50. Feature size 50 (lstm state has dimension of 50), because my sentence has 50 distinct characters, each character is encountered only once per epoch.
Performing backpropagation after 50 timesteps Using vanilla gradient descent, all fancy things like accelerated momentum or L2 norm, dropout are disabled.
I am sure there is no error, but interested if anyone can tell why Batch Norm has this stochastic property, if that's common with people.