Grad Checking, verify by average?

Question

I am running Gradient Checking, to spot any discrepancy between my mathematically-computed gradient and the actual sampled gradient - to reassure my backprop was implemented correctly.

When computing such a discrepancy, can I sum-up squares of differences, then take their average? I could then use this average as my estimate of how correctly the network computes the gradient:

$$\frac{1}{m}\sum_{i=0}^{i=m}(g_i-n_i)^2$$

or even:

$$\sqrt{\sum_{i=0}^{i=m}(g_i-n_i)^2}$$

where $g$ is a gradient from backpropagation, and $n$ is gradient from gradient checking.

---

However, Andrew Ng instead recommends:

$$\frac{\vert \vert (g-n) \vert \vert _2 }{ \vert \vert g \vert \vert _2 + \vert \vert n \vert \vert _2}$$

where $\vert \vert . \vert \vert _2$ is the length of the vector.

---

Another post post also recommends an slightly different approach: https://stats.stackexchange.com/a/188724/187816

---

Why would their approaches be better than mine?

Answer 1

Let me give you an example where Andrew's recommendation works better than yours:

Let's say that the real gradient is $(0, 0, 0)$ and the gradient you have computed is $(10^{-4}, 10^{-4}, 10^{-4})$. Then your average would return $10^{-8}$, and Andrew's recommendation would return $1$. Your metric could fool you into thinking that your gradient is computed propperly and the error is just due to a numeric issue, while Andrew's cannot fool you into that, due to the fact that it considers the fact that the gradient can be very small.

To wrap up, if your gradient doesn't have norm close to zero, it wouldn't really matter. However, when the gradient is close to zero you can be fooled into thinking that your gradient is right when it is not.