Deep fully connected NN with vanishing gradients

Question

I am making NN for choosing best bets possible for football matches. And I tried to make network quite deep (12 hidden layers with BN between them and ReLu as activation function) but it resulted in vanishing gradients problem.

Then I made it shallow (2-3 layers in same manner) and it resulted in great performance in train set but poor in validation set. My hypothesis is that in this configuration it could memorize all the possibilities (I have about 14 k of examples in train set).

Finally I made it in between this two - 8 hidden layers. And it resulted in quite nice performance both in train and val set. But i'm still a little worried about vanishing gradients and that it might be just a coincidence.

So I have few question in reference to what I've just written:

1. When should you be worried about vanishing gradients? I mean I can inspect some gradients distribution in tensorboard but this doesn't give me 100% sure answer.
2. Could using skip connections help?
3. Is there any rule how much layers is too much?

Answer 1

Vanishing gradients can occur because of adding so many layers to network. You can think neural networks as composite functions. During learning process gradients of loss function with respect to weights calculated according to chain rule, and because of large number of layers gradients can be so small by multiplications. It causes insufficient updates of weights towards optimum points of loss function. If your loss function barely decreasing, or there is no change at all then cause of this might be vanishing gradients.

Activation functions like tanh, sigmoid also can cause this problem, because their gradients become zero in edge points. Skip connections can ease vanishing gradient problem, because when connections are skipped gradient calculation follow a shorter path, and less multiplications can ease vanishing gradient problem. Resnets use skip connections to ease vanishing gradient problem: https://en.wikipedia.org/wiki/Residual_neural_network.

If you observe no decrease in loss function, then you can decrease number of layers gradually. It's a hyperparameter, and you can experiment with different numbers of layers to find optimum number which gives best results.