I am trying to train a deep network for twitter sentiment classification. It consists of an embedding layer (word2vec), an RNN (GRU) layer, followed by 2 conv layers, followed by 2 dense layers. Using ReLU for all activation functions.
I have just started using tensorboard & noticed that I seemingly have extremely small gradients in my convolutional layer weights (see figure)
I believe I have vanishing gradients since the distribution of CNN filter weights does not seem to change & the gradients are extremely small relative to the weights (see figure). [NOTE: the figure shows layer 1, but layers 2 looked very similar]
My questions are:
1) Am I interpreting the plots correctly that I do indeed have vanishing gradients & thus my convolutional layers arent learning? Does this mean they are currently essentially worthless?
2) What can I do to remedy this situation?
1) I have tried the network w/ just 1 layer and no layers (RNN-->FC), and having 2 kayers does empirically improve performance.
2) I have tried Xavier initialization and it doesnt do much (the previous default initialization mean value of .1 was very close to the Xander value)
3) By quick math, the gradients seem to change on the order of 1e-5, while the weights themselves are on the order of 1e-1. Thus at every iteration, the weights change 1e-5/1e-1*100% = ~.01%. Is this to be expected? What is the threshold for how much the weights change until we consider them to have converged / consider the changes to be useless in the sense that they dont change outcome?