2
$\begingroup$

I'm writing my own CNN code from scratch. Though I got fast, converged and satisfactory results, the trained weights change very little in value (while cost/loss function drops in time rapidly in a seemingly converged manner). My initial weights: convolution kernels as non zero unit matrices; fully connected layer weights as 0's. The activation function is sigmoid. The data scale from 0 to 1. Why do the weights change so little?

$\endgroup$
2
$\begingroup$

In machine learning, the vanishing gradient problem is a difficulty found in training artificial neural networks with gradient-based learning methods and backpropagation. In such methods, each of the neural network's weights receives an update proportional to the partial derivative of the error function with respect to the current weight in each iteration of training. The problem is that in some cases, the gradient will be vanishingly small, effectively preventing the weight from changing its value. In the worst case, this may completely stop the neural network from further training.

Source :: https://en.wikipedia.org/wiki/Vanishing_gradient_problem

Thus, gradient is vanishing till it reaches initial layer of neural network and in turn very little change in weights.

$\endgroup$
  • $\begingroup$ I agree with the reason, you (@feynman) can use (2D) Batch Normalization, which will normalize your batches not just at the input layer, but also in between all the layers you have. Moreover, since you (@feynman) told that you write your code from scratch, be pretty sure that you implement your forward and backprop in vectorized form rather than explicit for loop. Huge computational speed difference. $\endgroup$ – Ugur MULUK Feb 18 '19 at 11:44
  • $\begingroup$ @UgurMULUK thx. in fact i not only make sure vectorized form but also matricized form whenever necessary. esp when doing convolutions, matrices make those operations easier, so i dont flatten those matrices into vectors unless i have to. however, y does batch normalization resolve the vanishing gradient problem? $\endgroup$ – feynman Feb 18 '19 at 13:35
  • $\begingroup$ @Preet but i think vanishing gradient also means cost/function ceases to change? my cost function drops quickly tho $\endgroup$ – feynman Feb 18 '19 at 14:54
  • $\begingroup$ @feynman not exactly. the problem happens when the value of the derivate of your cost function is too small, as we multiply derivatives of cost functions in backpropagation. reference : ayearofai.com/… $\endgroup$ – Preet Feb 24 '19 at 17:55
1
$\begingroup$

I would venture that the problem you are having is at least due to bad initialisation, and it could also be a bad learning rate or activation functions.

You mentioned that you are initializing conv kernels to "non zero unit matrices" and "fully connected layer weights as 0's". Firstly I do not know what you mean by "non zero unit matrices", but definitely you should not be initializing fully connected layers to 0. The reason for this is because if all neurons in that layer have the same value, they will all behave very similar if not all exactly the same (depending on the network). This will produce very similar features that do not bias the network well.

I recommend using a random initialisation for both the Conv kernels and Dense kernels, and zeros for any biases.

Secondly, your activation functions are not stated, but if you are using an activation function with a gradient that is sensitive to learning rate vanishing, then this is also a problem. For instance using a sigmoid activation and a high learning rate will result in almost no weight update value.

Also, what do you consider [why the weights change] "so little"? I would expect for data scaled from [-1, 1] the update magnitude should be around 1e-3 for the first layers and 1e-4 for the last ones. (Off the top of my head)

$\endgroup$
  • $\begingroup$ thx. i changed init weights to b all random.but the situation is unchanged. my activation function is sigmoid. my data scale from 0 to 1. my cost function drops exponentially when weights keep changing even when cost function hardly changes $\endgroup$ – feynman Feb 19 '19 at 5:12
  • $\begingroup$ What are your intermediate layer activations, and your output layer activation, the size of the output units, the dimension of the labels (excluding batch), your learning rate and optimizer? $\endgroup$ – Gouda Feb 19 '19 at 5:23
  • $\begingroup$ there's no activation function on my intermediate/convolution layer. the output layer activation is sigmoid. my learning rate is self adapting, quickened when cost function drops quickly and vice versa. what do u mean by output unit size and dimension of labels? $\endgroup$ – feynman Feb 19 '19 at 6:12
0
$\begingroup$

@feynman Because Batch Normalization do not let weights vanish or explode, it normalizes the batches in between every layer. Since each activation function will have inputs from the previous layers as close to zero, the effect of the increasing/decreasing weights of the previous layers will be suppressed; avoiding the snowball effect.

$\endgroup$
  • $\begingroup$ i think an activation function is needed on the final layer. is it ok to not use it on previous layers, if i dont find it necessary? $\endgroup$ – feynman Feb 19 '19 at 2:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.