I'm trying to understand how to feed data into LSTM layer of Keras, but I'm in trouble and I don't understand how to do it. I've have a dataset composed by words and each words is embedded with a vector of 839 elements, so the shape of my dataset is (x, 839).

I want to feed my dataset into the LSTM layer, but I don't properly understand the 3D object wanted by Keras, composed by (batch_size, timesteps, feature). I want to feed one word per time into the LSTM, how can I do?

Thanks in advance.

UPDATE I still receive an error about input shape:

ValueError: Error when checking input: expected lstm_11_input to have shape (2, 839) but got array with shape (839, 1)

I'm using batch_size_shape(batch_size, timesteps, feature) at the moment. This is the code:

class KendallTauHistory(Callback):
  def __init__(self, dataset, y_true, groups):
    self.y_true = y_true
    self.dataset = dataset
    self.groups = groups

  def on_epoch_end(self, epoch, logs=None):
    predictions = self.model.predict(self.dataset)
    predictions = predictions.flatten()
    predictions = list(map(lambda element: element + np.random.uniform(0.0, 1.0) * 0.02 - 0.01, predictions))
    # For batch training
    ranked_predictions = np.array([])
    kendalls = np.array([])
    start_range = 0
    for group in self.groups:
        end_range = (start_range + group[1]) # Batch is a group of words with same group id
        batch_predictions = predictions[start_range:end_range]
        batch_labels = self.y_true[start_range:end_range]
        batch_predictions = list(map(lambda element: element + np.random.uniform(0.0, 1.0) * 0.02 - 0.01, batch_predictions))
        ranked_predictions = np.append(ranked_predictions, np.floor(rankdata(batch_predictions)))
        kendalls = np.append(kendalls, kendalltau(batch_labels, batch_predictions))
        start_range = end_range
    #self.y_true = self.y_true[0:len(ranked_predictions)]
    print('\nORIGINAL LABELS: {0}\n'.format(self.y_true))
    print('PREDICTED LABELS: {0}'.format(ranked_predictions))
    print("\nEpoch Kendall's tau: {0}".format(np.mean(kendalls)))

model = tf.keras.Sequential()
model.add(LSTM(units=10, batch_input_shape=(None, 2, 839)))
model.add(Dense(15, activation='sigmoid'))


model.compile(loss=listnet_loss, optimizer=keras.optimizers.Nadam(learning_rate=0.000005, beta_1=0.9, beta_2=0.999))
real_labels = np.array([])
losses = np.array([])

with tf.device('/GPU:0'):
  model.fit(training_dataset, training_dataset_labels, epochs=10, workers=10,
            verbose=1, callbacks=[KendallTauHistory(training_dataset, training_dataset_labels, groups_id_count)])

Take the example of time series: $\mathbf{x}1,\mathbf{x}2,\ldots,\mathbf{x}10$ where each $\mathbf{x}i$ is let 5 dimensional. The 'timestep' here will be the window chosen such that value at time instant is dependent on previous $p$ lags. So data passed to LSTM will be of the form $\mathbf{x}1,\mathbf{x}2$ as input with $p=2$ lags. In your case as well, each word is dependent on previous words. So timestep will be the number of previous words you need to pass to LSTM in context of current words. Batchsize argument is the number of examples after which the backpropagation will happen. It is a free parameter you can control it but it should be completely divisible by number of training examples. Feature will be 839, the length of embedding vector.

| improve this answer | |
  • $\begingroup$ thank you for the explanation. I'll try this :) $\endgroup$ – pairon Feb 23 at 18:26
  • $\begingroup$ If I create the LSTM layer as follow: model.add(LSTM(units=100, input_shape=(2, 839))) I receive the following error: Error when checking input: expected lstm_input to have shape (2, 1) but got array with shape (839, 1). $\endgroup$ – pairon Feb 23 at 20:28
  • $\begingroup$ @pairon I suggest to use batch_input_shape=(batch_size, timesteps, feature) in your first layer. $\endgroup$ – shaifali Gupta Feb 24 at 7:43
  • $\begingroup$ I used it but I still receive the error. I update my answer description with the code so also you can understand better. $\endgroup$ – pairon Feb 24 at 10:40

Input data of LSTM() layers must follow this pattern:

( Number of observations , Window size , Number of input series )

Number of observations is the size of your mini batch; Window size is the length of each input series, another hyperparameter you can choose; Number of input series is the number of explanatory variables you are using.

| improve this answer | |
  • $\begingroup$ thank you for the answer. I still receive error, I updated my answer description with the error and code. $\endgroup$ – pairon Feb 24 at 10:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.