Gradient descent formula

Question

I came across an interesting book about neural network basics, and the formula for gradient descent from one of the first chapters says:

Gradient descent: For each layer update the weights according to the rule

$w^l \rightarrow w^l-\frac{\eta}{m} \sum_x \delta^{x,l} (a^{x,l-1})^T$

where $w^l$ is the weights matrix in layer $l$, and $x$ is the index of a specific training example.

I don't want to rewrite all formulas from the chapter, but the important part one is BP4 - equation for the rate of change of the cost with respect to any weight in the network:

$\frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j$

Am I missing something or the first formula is incorrect? Shouldn't we use Hadamard product instead, like this?

$w^l \rightarrow w^l-\frac{\eta}{m} \sum_x \delta^{x,l} \odot a^{x,l-1}$

Thanks for help.

Answer 1

You can see the result by writing down what you get explicitly in both cases. What you want to get after summation in the update rule is:

$\sum_x \delta^{x,l} (a^{x,l-1})^T = \delta^{1,l} a^{1,l-1}+\delta^{2,l} a^{2,l-1}+...$

which is obtained by the form given in the text.

Keep in mind that Hadamard product can be done only for vectors of same dimension and result in another vector of the same dimension. I suppose you can treat $\delta$ as an $m$ dimensional vector (m being number of training examples) with vector components and take a Hadamard product of it with $a$, but you will end up with another $m$ dimensional vector. And at that point, summing over $x$ becomes meaningless. You can say "Do a Hadamard product and then sum the rows", and this will give the above result.

Edit:

I will try to further clarify the dimensions of the objects given. Let us assume that the layer l has j neurons, and the layer l-1 has k neurons. That makes $w^l$ an $(j x k)$ dimensional matrix. The update equation has to conserve the dimensionality, so the right hand side also has to be matrix of the same size.

Now, $a^{(l-1)}$ is a vector with k components, corresponding to the activations from the neurons of the layer l-1. Likewise, the error of the layer l has j components, one for each neuron.

To get a $(j x k)$ matrix, we take the outer product of these two vectors, that is why there is a transpose in the equation. And finally we sum over all training samples to finalize the calculation.