RNN: why Wx + Uh instead of W[x,h]

Traditionally, a state for RNN is computed as $$h_t = \sigma(W\cdot \vec x + U\cdot \vec h_{t-1} + \vec b)$$

For a RNN, why to add-up the terms $(Wx + Uh_{t-1})$ instead of just having a single matrix times a concatenated vector:

$$W_m[x, h_{t-1}]$$

where $[...]$ is concatenation.

In other words, we would end up with a long vector like $\{x_1, x_2, x_3, h_{1,t-1}, h_{2,t-1}, h_{3,t-1} \}$ multiplied by $W_m$.

It seems like the second approach would have a significantly larger matrix, which has more elements than $W$ and $U$ combined.

Does that mean $W$ and $U$ are a simplification, what do we lose by using them, and adding up the results?

• slapping -1 with no explanation given on how to improve, thanks! :D – Kari Apr 12 '18 at 9:47
• Its one of downsides of our community that I don't know why happens. Don't get disappointed :) – Media Apr 12 '18 at 10:30

Theoretically, the formula with two matrices is more clear and self-evident, I think that's the reason why it's used more often. In practice, both approaches are actually used in production and hence are equivalent. It's just a matter of preference.

Tensorflow

For example, Tensorflow is often optimized for performance. Here's how basic RNN cell is implemented there (tensorflow/python/ops/rnn_cell_impl.py):

def call(self, inputs, state):
"""Most basic RNN: output = new_state = act(W * input + U * state + B)."""

gate_inputs = math_ops.matmul(
array_ops.concat([inputs, state], 1), self._kernel)
gate_inputs = nn_ops.bias_add(gate_inputs, self._bias)
output = self._activation(gate_inputs)
return output, output

A single matrix multiplication is more efficient, so it's applied, even though the comment describes the expanded formula with two matrices.

Keras

On the other hand, Keras often chooses simplicity and clarity over performance. Here's its implementation (keras/layers/recurrent.py):

def call(self, inputs, states, training=None):
prev_output = states

...

if dp_mask is not None:
h = K.dot(inputs * dp_mask, self.kernel)
else:
h = K.dot(inputs, self.kernel)
if self.bias is not None:
h = K.bias_add(h, self.bias)

...
output = h + K.dot(prev_output, self.recurrent_kernel)
if self.activation is not None:
output = self.activation(output)

The class thus makes its it easy to access two matrices separately (self.kernel and self.recurrent_kernel).

Pytorch

Pytorch approach is closer to keras. Moreover, not only they use separate kernel matrices, they also have two bias vectors, and these four arrays are accessible for the client code.

Despite these differences all three libraries are functionally equivalent.