I am trying to understand the backpropagation algorithm in a multi-layer perceptron environment.

Algorithm 6.4 Backward computation for the deep neural network of algorithm 6.3, which uses, in addition to the input $\boldsymbol{x}$, a target $\boldsymbol{y}$. This computation yields the gradients on the activations $\boldsymbol{a}^{(k)}$ for each layer $k$, starting from the output layer and going backwards to the first hidden layer. From these gradients, which can be interpreted as an indication of how each layer's output should change to reduce error, one can obtain the gradient on the parameters of each layer. The gradients on weights and biases can be immediately used as part of a stochastic gradient update (performing the update right after the gradients have been computed) or used with other gradient-based optimization methods. After the forward computation, compute the gradient on the output layer:

${g} \leftarrow \nabla_{\hat{y}} J=\nabla_{\hat{y}} L(\hat{y}, > y)1$
for$k$ from $k=l, l-1, ...$ to$1$
Convert the gradient on the layer's output into a gradient on the prenonlinearity activation (element-wise multiplication if $f$ is element-wise):
${g} \leftarrow > \nabla_{{a}^{(k)}} J={g} \odot f^{\prime}\left({a}^{(k)}\right)$
Compute gradients on weights and biases (including the regularization term, where needed):
$\nabla_{{b}^{(k)}} J={g}+\lambda > \nabla_{{b}^{(k)}}\Omega(\theta)$
$\nabla_{{W}^{(k)}} J={g} ;{h}^{(k-1) > \top}+\lambda \nabla_{{W}^{(k)}}$<--- HERE
Propagate the gradients w.r.t. the next lower-level hidden layer's activations:
${g} \leftarrow > \nabla_{{h}^{(k-1)}} J={W}^{(k) \top};{g}$ <--- HERE

Bengio and al., Deep Learning

I struggle to understand what ";" means in the algorithm. Is the semicolon (;) notation used to indicate that operations are performed concurrently (element-wise)like performing an element-by-element multiplication between the gradient g and the former activation layer h(k-1)? But shouldn't it be $\odot$?

I wonder if Bengi and al. aren't complicating the algorithm ... I am a slow learner in mathematics, so don't hesitate to use easy concepts ;)



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.