I'm stuck regarding the implementation of backpropagation in a binary classification task using a GRU. I wanted to know if someone could tell me how to proceed. I was able to understand how BP works in a simple RNN because it is very similar to an MLP. However, I don't know how to do it in the context of a GRU.

Let's say we want to use a GRU RNN for our binary classification task, i.e. $y_{true} \in \{0, 1\}$.

We have the following Gated recurrent unit

\begin{equation*} \begin{aligned} \Gamma_{r}^{<t>} & = \sigma\left(\gamma_{r}^{<t>}\right) = \sigma\left(U_r \; a^{<t-1>} + W_r \; x^{<t>} + b_r\right) \\ \\ \Gamma_{u}^{<t>} & = \sigma\left(\gamma_{u}^{<t>}\right) = \sigma\left(U_u \; a^{<t-1>} + W_u \; x^{<t>} + b_u\right) \\ \\ C^{<t>} & = tanh\left(c^{<t>}\right) = tanh\left(U_c \left(\Gamma_{r}^{<t>} \odot a^{<t-1>} \right) + W_c \; x^{<t>} + b_c\right) \\ \\ a^{<t>} & = \left(1-\Gamma_{u}^{<t>}\right) \odot a^{<t-1>} + \Gamma_{u}^{<t>} \odot C^{<t>} \end{aligned} \end{equation*}

Only at the final time $T$, we compute

\begin{equation*} \begin{split} v^{<T>} & = W_{ya} a^{<T>}+b_{y}, \\ \\ y^{<T>} & = \sigma\left(v^{<T>}\right). \end{split} \end{equation*}

Hence, the loss function is only computed at the final time $T$ over a single training example as

$$ \mathcal{L} = - \left[y_{true} \times ln \left(y^{<T>}\right) + \left(1-y_{true}\right) \times ln \left(1-y^{<T>}\right)\right]. $$

The activation functions are $$ \sigma(x) = \frac{1}{1+e^{-x}} $$ $$ tanh(x) = \frac{e^{x}-e^{-x}}{e^{x}+e^{-x}} $$

The question is, how can I compute the derivatives w.r.t. all the parameters?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.