1
$\begingroup$

I trained a Bidirectional LSTM of width 384 which took in an input of dimension (None, 4995, 12), using Tensorflow/Keras. I've been trying to understand how my input dimension is mapped to the $x_t$ in an LSTM cell input because I notice when I examine the internal weights of my LSTM that my weights are of dimension (384,12).

Does Tensorflow/Keras automatically divide up my input array into partitions of size 384? I assume that it doesn't time-slice my input because that is a feature of TimeDistributed LSTMs. Any information about this would be helpful.

UPDATE: I discovered when perusing a Tensorflow implementation of an LSTM that the input data is reshaped to be the same length as the $x_t$ array. My question, then, is whether the Tensorflow/Keras backend is implemented the same way.

def lstm(inputs, state, params):
    W_xi, W_hi, b_i, W_xf, W_hf, b_f, W_xo, W_ho, b_o, W_xc, W_hc, b_c, W_hq, b_q = params
    (H, C) = state
    outputs = []
    for X in inputs:
        X = tf.reshape(X, [-1, W_xi.shape[0]])
        I = tf.sigmoid(tf.matmul(X, W_xi) + tf.matmul(H, W_hi) + b_i)
        F = tf.sigmoid(tf.matmul(X, W_xf) + tf.matmul(H, W_hf) + b_f)
        O = tf.sigmoid(tf.matmul(X, W_xo) + tf.matmul(H, W_ho) + b_o)
        C_tilda = tf.tanh(tf.matmul(X, W_xc) + tf.matmul(H, W_hc) + b_c)
        C = F * C + I * C_tilda
        H = O * tf.tanh(C)
        Y = tf.matmul(H, W_hq) + b_q
        outputs.append(Y)
    return tf.concat(outputs, axis=0), (H, C)
$\endgroup$
2
  • $\begingroup$ Are you still looking for an answer to this? $\endgroup$
    – hH1sG0n3
    Commented Jun 22, 2022 at 9:52
  • 1
    $\begingroup$ @hH1sG0n3 I think I understand LSTMs much better now, as indicated below. No additional answer needed unless you have a clarification $\endgroup$ Commented Jul 5, 2022 at 20:11

1 Answer 1

1
$\begingroup$

The output of an LSTM is of the shape $[?,t,d]$ where $t$ is the number of timesteps and $d$ is the hidden dimension and there are $t$ number of cells in the LSTM - unless "return_sequences" is set to false in Keras. When "return_sequences" is false, the output is $[?,d]$ because it is returning the final timestep of what the output would have been if "return_sequences" had been set to true. This means that the input array is not being partitioned in any way - 384 was the hidden dimension.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.