2
$\begingroup$

In lecture notes for cs231n while discussing checking analytical gradient with numerical gradient the paragraph says this:

Check only few dimensions. In practice the gradients can have sizes of million parameters. In these cases it is only practical to check some of the dimensions of the gradient and assume that the others are correct.

Be careful: One issue to be careful with is to make sure to gradient check a few dimensions for every separate parameter. In some applications, people combine the parameters into a single large parameter vector for convenience. In these cases, for example, the biases could only take up a tiny number of parameters from the whole vector, so it is important to not sample at random but to take this into account and check that all parameters receive the correct gradients.

I understand to check only a few dimensions but what does the part after be careful mean. I fail to understand that warning. can somebody explain it?

$\endgroup$
1
$\begingroup$

As you might know, biases are used in the CNN only when there is no batch normalization layer is included because this automatically adds a type of bias to the activation outputs.

So, this is what I understood:

Let us assume a layer of CNN (without batch norm) with 50 kernels each of size of 5 x 5. So, total weight updates will be 5 x 5 x 50 = 1250 and total biases updates will be 50 (a bias for each kernel).

For the part "...people combine the parameters into a single large parameter vector for convenience. In these cases, for example, the biases could only take up a tiny number of parameters from the whole vector..." :-

  • If you merge all these parameters into one single vector and randomly sample the parameters to check gradients for, you may end up checking very few or none of the gradient updates for biases as they make a very less portion of that vector (only 50 out of 1300).

For the part "One issue to be careful with is to make sure to gradient check a few dimensions for every separate parameter..." :-

  • On the other hand, you can choose some kernels and take all parameters (weights and biases) of only those kernels to check gradient for and so in this case you will definitely be checking gradients of some biases which they say is a good way of doing gradient checking.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.