0
$\begingroup$

I'm stuck regarding the implementation of backpropagation in a binary classification task using a GRU. I wanted to know if someone could tell me how to proceed. I was able to understand how BP works in a simple RNN because it is very similar to an MLP. However, I don't know how to do it in the context of a GRU.



Let's say we want to use a GRU RNN for our binary classification task, i.e. $y_{true} \in \{0, 1\}$.

We have the following Gated recurrent unit

\begin{equation*} \begin{aligned} \Gamma_{r}^{<t>} & = \sigma\left(\gamma_{r}^{<t>}\right) = \sigma\left(U_r \; a^{<t-1>} + W_r \; x^{<t>} + b_r\right) \\ \\ \Gamma_{u}^{<t>} & = \sigma\left(\gamma_{u}^{<t>}\right) = \sigma\left(U_u \; a^{<t-1>} + W_u \; x^{<t>} + b_u\right) \\ \\ C^{<t>} & = tanh\left(c^{<t>}\right) = tanh\left(U_c \left(\Gamma_{r}^{<t>} \odot a^{<t-1>} \right) + W_c \; x^{<t>} + b_c\right) \\ \\ a^{<t>} & = \left(1-\Gamma_{u}^{<t>}\right) \odot a^{<t-1>} + \Gamma_{u}^{<t>} \odot C^{<t>} \end{aligned} \end{equation*}



Only at the final time $T$, we compute

\begin{equation*} \begin{split} v^{<T>} & = W_{ya} a^{<T>}+b_{y}, \\ \\ y^{<T>} & = \sigma\left(v^{<T>}\right). \end{split} \end{equation*}

Hence, the loss function is only computed at the final time $T$ over a single training example as

$$ \mathcal{L} = - \left[y_{true} \times ln \left(y^{<T>}\right) + \left(1-y_{true}\right) \times ln \left(1-y^{<T>}\right)\right]. $$



The activation functions are $$ \sigma(x) = \frac{1}{1+e^{-x}} $$ $$ tanh(x) = \frac{e^{x}-e^{-x}}{e^{x}+e^{-x}} $$



The question is, how can I compute the derivatives w.r.t. all the parameters?

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.