I am trying to train a deep network for twitter sentiment classification. It consists of an embedding layer (word2vec), an RNN (GRU) layer, followed by 2 conv layers, followed by 2 dense layers. Using ReLU for all activation functions.

I have just started using tensorboard & noticed that I seemingly have extremely small gradients in my convolutional layer weights (see figure)

enter image description here

I believe I have vanishing gradients since the distribution of CNN filter weights does not seem to change & the gradients are extremely small relative to the weights (see figure). [NOTE: the figure shows layer 1, but layers 2 looked very similar]

My questions are:

1) Am I interpreting the plots correctly that I do indeed have vanishing gradients & thus my convolutional layers arent learning? Does this mean they are currently essentially worthless?

2) What can I do to remedy this situation?


UPDATE 3/13/18

Few comments:

1) I have tried the network w/ just 1 layer and no layers (RNN-->FC), and having 2 kayers does empirically improve performance.

2) I have tried Xavier initialization and it doesnt do much (the previous default initialization mean value of .1 was very close to the Xander value)

3) By quick math, the gradients seem to change on the order of 1e-5, while the weights themselves are on the order of 1e-1. Thus at every iteration, the weights change 1e-5/1e-1*100% = ~.01%. Is this to be expected? What is the threshold for how much the weights change until we consider them to have converged / consider the changes to be useless in the sense that they dont change outcome?

  • $\begingroup$ It’s hard to say they have vanished vs they have converged. Are you getting reasonable performance? Does the loss continue to decrease? $\endgroup$ – kbrose Mar 9 '18 at 14:47

In order to fix the problem of vanishing gradients, you can use Xavier Initilization. Also, the implementation of Xavier Initialization in tensorflow can be done by following this thread.


Edit: Definitely try Xavier initialization first, as the other answerer said.

In other cases, where you have to increase the gradient manually...

Gradient means rate of change of your loss function. If your loss function is not changing much with respect to certain weights, then changing those weights doesn't change your loss function.

The weights just determine the type of linear combination from the previous layer. If the loss doesn't change when you change the linear combination, then you need to amplify the effect of increasing or decreasing the linear combination. So what you want is, whenever the weights increase, you want the linear combination to increase by more, and whenever the weights decrease, you want the linear combination to decrease by more.

Let's say you multiplied a weight by a constant k. Then if you increased that weight the linear combination would increase more, and if you decreased the weight then the linear combination would decrease more. So it must be that multiplying your weights by a constant k > 1 would increase the effects on the linear combination from changing the weights.

If you multiply all your weights by a constant k, then that's the same thing as multiplying the entire linear combination by k. So you want to multiply the linear combination by k before squishing it with your activation function.

Your new activation function then would be this:

activation = ReLU( linear combination x k )


  • activation = 0 x (linear combination x k), if linear combination <= 0

  • activation = 1 x (linear combination x k), if linear combination > 0

Compare to regular ReLU, which is:

  • activation = 0 x linear combination, if linear combination <= 0

  • activation = 1 x linear combination, if linear combination > 0

So you can see that no matter what the case is, you want to multiply the ReLU activation by k. Specifically, you want to multiply the ReLU activation by k > 1.

In practical terms, this increases the slope of the right side of the ReLU function.

  • $\begingroup$ Thanks @default. Another interpretation of this solution is specifying a unique learning rate for each ReLU layer rather than a global learning rate. This is definitely worth trying if nothing else works, but it does increase the parameter space. Xander initialization is a principled way of algorithmically standardizing the k parameter so one doesn't have to do it by hand... $\endgroup$ – DankMasterDan Mar 9 '18 at 19:13
  • $\begingroup$ Xavier initialization is definitely the first thing you should try. You can think of it as such: if you have 10 different inputs and you take a tenth of each, then the result will be the same size as those inputs, but if you take a thousandth of each then the result will be a hundredth the size of those inputs. If the latter layers do this, then the previous layers will have a very small magnitude of influence on the final output since it will be hundreds of times smaller. @DankMasterDan $\endgroup$ – Default picture Mar 9 '18 at 20:07

You also could try to use a Batch Norm layer. It normalizes the outputs of previous layer, which prevents gradients from being too small - see detailed explanation here


Activation functions

What activation function are you using?

The most common solution to the vanishing gradient issue in deep networks is to use activation functions that don't have this problem, i.e ReLU instead of sigmoid or tanh activation.

  • $\begingroup$ I am using ReLU already (edited question to reflect this) $\endgroup$ – DankMasterDan Mar 8 '18 at 23:33

After studying about vanishing gradient problem I found some solutions including the already answered ones that are listed below:-
1- Relu activation function
2- Greedy layer-wise pre-training
3- Using weight initializers
4- using chunk like architecture as used in Resnet, SEresnet where the architecture is divided into 3 chunks and in all of the inputs are

  • previous chunk output
  • the initial input to NN

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.