How to fix these vanishing gradients?

Question

I am trying to train a deep network for twitter sentiment classification. It consists of an embedding layer (word2vec), an RNN (GRU) layer, followed by 2 conv layers, followed by 2 dense layers. Using ReLU for all activation functions.

I have just started using tensorboard & noticed that I seemingly have extremely small gradients in my convolutional layer weights (see figure)

I believe I have vanishing gradients since the distribution of CNN filter weights does not seem to change & the gradients are extremely small relative to the weights (see figure). [NOTE: the figure shows layer 1, but layers 2 looked very similar]

My questions are:

1) Am I interpreting the plots correctly that I do indeed have vanishing gradients & thus my convolutional layers arent learning? Does this mean they are currently essentially worthless?

2) What can I do to remedy this situation?

Thanks!

UPDATE 3/13/18

Few comments:

1) I have tried the network w/ just 1 layer and no layers (RNN-->FC), and having 2 kayers does empirically improve performance.

2) I have tried Xavier initialization and it doesnt do much (the previous default initialization mean value of .1 was very close to the Xander value)

3) By quick math, the gradients seem to change on the order of 1e-5, while the weights themselves are on the order of 1e-1. Thus at every iteration, the weights change 1e-5/1e-1*100% = ~.01%. Is this to be expected? What is the threshold for how much the weights change until we consider them to have converged / consider the changes to be useless in the sense that they dont change outcome?

Answer 1

In order to fix the problem of vanishing gradients, you can use Xavier Initilization. Also, the implementation of Xavier Initialization in tensorflow can be done by following this thread.

Answer 2

You also could try to use a Batch Norm layer. It normalizes the outputs of previous layer, which prevents gradients from being too small - see detailed explanation here

Answer 3

Edit: Definitely try Xavier initialization first, as the other answerer said.

In other cases, where you have to increase the gradient manually...

Gradient means rate of change of your loss function. If your loss function is not changing much with respect to certain weights, then changing those weights doesn't change your loss function.

The weights just determine the type of linear combination from the previous layer. If the loss doesn't change when you change the linear combination, then you need to amplify the effect of increasing or decreasing the linear combination. So what you want is, whenever the weights increase, you want the linear combination to increase by more, and whenever the weights decrease, you want the linear combination to decrease by more.

Let's say you multiplied a weight by a constant k. Then if you increased that weight the linear combination would increase more, and if you decreased the weight then the linear combination would decrease more. So it must be that multiplying your weights by a constant k > 1 would increase the effects on the linear combination from changing the weights.

If you multiply all your weights by a constant k, then that's the same thing as multiplying the entire linear combination by k. So you want to multiply the linear combination by k before squishing it with your activation function.

Your new activation function then would be this:

activation = ReLU( linear combination x k )

or

• activation = 0 x (linear combination x k), if linear combination <= 0

• activation = 1 x (linear combination x k), if linear combination > 0

Compare to regular ReLU, which is:

• activation = 0 x linear combination, if linear combination <= 0

• activation = 1 x linear combination, if linear combination > 0

So you can see that no matter what the case is, you want to multiply the ReLU activation by k. Specifically, you want to multiply the ReLU activation by k > 1.

In practical terms, this increases the slope of the right side of the ReLU function.

Answer 4

Activation functions

What activation function are you using?

The most common solution to the vanishing gradient issue in deep networks is to use activation functions that don't have this problem, i.e ReLU instead of sigmoid or tanh activation.

Answer 5

After studying about vanishing gradient problem I found some solutions including the already answered ones that are listed below:-
1- Relu activation function
2- Greedy layer-wise pre-training
3- Using weight initializers
4- using chunk like architecture as used in Resnet, SEresnet where the architecture is divided into 3 chunks and in all of the inputs are

• previous chunk output
• the initial input to NN