I am making NN for choosing best bets possible for football matches. And I tried to make network quite deep (12 hidden layers with BN between them and ReLu as activation function) but it resulted in vanishing gradients problem.
Then I made it shallow (2-3 layers in same manner) and it resulted in great performance in train set but poor in validation set. My hypothesis is that in this configuration it could memorize all the possibilities (I have about 14 k of examples in train set).
Finally I made it in between this two - 8 hidden layers. And it resulted in quite nice performance both in train and val set. But i'm still a little worried about vanishing gradients and that it might be just a coincidence.
So I have few question in reference to what I've just written:
- When should you be worried about vanishing gradients? I mean I can inspect some gradients distribution in tensorboard but this doesn't give me 100% sure answer.
- Could using skip connections help?
- Is there any rule how much layers is too much?