Theoretically, the formula with two matrices is more clear and self-evident, I think that's the reason why it's used more often.
In practice, both approaches are actually used in production and hence are equivalent. It's just a matter of preference.
For example, Tensorflow is often optimized for performance. Here's how basic RNN cell is implemented there (
def call(self, inputs, state):
"""Most basic RNN: output = new_state = act(W * input + U * state + B)."""
gate_inputs = math_ops.matmul(
array_ops.concat([inputs, state], 1), self._kernel)
gate_inputs = nn_ops.bias_add(gate_inputs, self._bias)
output = self._activation(gate_inputs)
return output, output
A single matrix multiplication is more efficient, so it's applied, even though the comment describes the expanded formula with two matrices.
On the other hand, Keras often chooses simplicity and clarity over performance. Here's its implementation (
def call(self, inputs, states, training=None):
prev_output = states
if dp_mask is not None:
h = K.dot(inputs * dp_mask, self.kernel)
h = K.dot(inputs, self.kernel)
if self.bias is not None:
h = K.bias_add(h, self.bias)
output = h + K.dot(prev_output, self.recurrent_kernel)
if self.activation is not None:
output = self.activation(output)
The class thus makes its it easy to access two matrices separately (
Pytorch approach is closer to keras. Moreover, not only they use separate kernel matrices, they also have two bias vectors, and these four arrays are accessible for the client code.
Despite these differences all three libraries are functionally equivalent.