Traditionally, a state for RNN is computed as $$h_t = \sigma(W\cdot \vec x + U\cdot \vec h_{t-1} + \vec b)$$

For a RNN, why to add-up the terms $(Wx + Uh_{t-1})$ instead of just having a single matrix times a concatenated vector:

$$W_m[x, h_{t-1}]$$

where $[...]$ is concatenation.

In other words, we would end up with a long vector like $\{x_1, x_2, x_3, h_{1,t-1}, h_{2,t-1}, h_{3,t-1} \}$ multiplied by $W_m$.

It seems like the second approach would have a significantly larger matrix, which has more elements than $W$ and $U$ combined.

Does that mean $W$ and $U$ are a simplification, what do we lose by using them, and adding up the results?

  • $\begingroup$ slapping -1 with no explanation given on how to improve, thanks! :D $\endgroup$ – Kari Apr 12 '18 at 9:47
  • 2
    $\begingroup$ Its one of downsides of our community that I don't know why happens. Don't get disappointed :) $\endgroup$ – Media Apr 12 '18 at 10:30

Theoretically, the formula with two matrices is more clear and self-evident, I think that's the reason why it's used more often. In practice, both approaches are actually used in production and hence are equivalent. It's just a matter of preference.


For example, Tensorflow is often optimized for performance. Here's how basic RNN cell is implemented there (tensorflow/python/ops/rnn_cell_impl.py):

def call(self, inputs, state):
    """Most basic RNN: output = new_state = act(W * input + U * state + B)."""

    gate_inputs = math_ops.matmul(
        array_ops.concat([inputs, state], 1), self._kernel)
    gate_inputs = nn_ops.bias_add(gate_inputs, self._bias)
    output = self._activation(gate_inputs)
    return output, output

A single matrix multiplication is more efficient, so it's applied, even though the comment describes the expanded formula with two matrices.


On the other hand, Keras often chooses simplicity and clarity over performance. Here's its implementation (keras/layers/recurrent.py):

def call(self, inputs, states, training=None):
    prev_output = states[0]


    if dp_mask is not None:
        h = K.dot(inputs * dp_mask, self.kernel)
        h = K.dot(inputs, self.kernel)
    if self.bias is not None:
        h = K.bias_add(h, self.bias)

    output = h + K.dot(prev_output, self.recurrent_kernel)
    if self.activation is not None:
        output = self.activation(output)

The class thus makes its it easy to access two matrices separately (self.kernel and self.recurrent_kernel).


Pytorch approach is closer to keras. Moreover, not only they use separate kernel matrices, they also have two bias vectors, and these four arrays are accessible for the client code.

Despite these differences all three libraries are functionally equivalent.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.