Feed data into Keras LSTM layer

Question

I'm trying to understand how to feed data into LSTM layer of Keras, but I'm in trouble and I don't understand how to do it. I've have a dataset composed by words and each words is embedded with a vector of 839 elements, so the shape of my dataset is (x, 839).

I want to feed my dataset into the LSTM layer, but I don't properly understand the 3D object wanted by Keras, composed by (batch_size, timesteps, feature). I want to feed one word per time into the LSTM, how can I do?

Thanks in advance.

UPDATE I still receive an error about input shape:

ValueError: Error when checking input: expected lstm_11_input to have shape (2, 839) but got array with shape (839, 1)

I'm using batch_size_shape(batch_size, timesteps, feature) at the moment. This is the code:

class KendallTauHistory(Callback):
  def __init__(self, dataset, y_true, groups):
    self.y_true = y_true
    self.dataset = dataset
    self.groups = groups

  def on_epoch_end(self, epoch, logs=None):
    predictions = self.model.predict(self.dataset)
    predictions = predictions.flatten()
    predictions = list(map(lambda element: element + np.random.uniform(0.0, 1.0) * 0.02 - 0.01, predictions))
    # For batch training
    ranked_predictions = np.array([])
    kendalls = np.array([])
    start_range = 0
    for group in self.groups:
        end_range = (start_range + group[1]) # Batch is a group of words with same group id
        batch_predictions = predictions[start_range:end_range]
        batch_labels = self.y_true[start_range:end_range]
        batch_predictions = list(map(lambda element: element + np.random.uniform(0.0, 1.0) * 0.02 - 0.01, batch_predictions))
        ranked_predictions = np.append(ranked_predictions, np.floor(rankdata(batch_predictions)))
        kendalls = np.append(kendalls, kendalltau(batch_labels, batch_predictions))
        start_range = end_range
    #self.y_true = self.y_true[0:len(ranked_predictions)]
    print('\nORIGINAL LABELS: {0}\n'.format(self.y_true))
    print('PREDICTED LABELS: {0}'.format(ranked_predictions))
    print("\nEpoch Kendall's tau: {0}".format(np.mean(kendalls)))


model = tf.keras.Sequential()
model.add(LSTM(units=10, batch_input_shape=(None, 2, 839)))
model.add(Dense(15, activation='sigmoid'))

model.summary()

model.compile(loss=listnet_loss, optimizer=keras.optimizers.Nadam(learning_rate=0.000005, beta_1=0.9, beta_2=0.999))
real_labels = np.array([])
losses = np.array([])

with tf.device('/GPU:0'):
  model.fit(training_dataset, training_dataset_labels, epochs=10, workers=10,
            verbose=1, callbacks=[KendallTauHistory(training_dataset, training_dataset_labels, groups_id_count)])

Answer 1

Take the example of time series: $\mathbf{x}1,\mathbf{x}2,\ldots,\mathbf{x}10$ where each $\mathbf{x}i$ is let 5 dimensional. The 'timestep' here will be the window chosen such that value at time instant is dependent on previous $p$ lags. So data passed to LSTM will be of the form $\mathbf{x}1,\mathbf{x}2$ as input with $p=2$ lags. In your case as well, each word is dependent on previous words. So timestep will be the number of previous words you need to pass to LSTM in context of current words. Batchsize argument is the number of examples after which the backpropagation will happen. It is a free parameter you can control it but it should be completely divisible by number of training examples. Feature will be 839, the length of embedding vector.

Answer 2

Input data of LSTM() layers must follow this pattern:

( Number of observations , Window size , Number of input series )

Number of observations is the size of your mini batch; Window size is the length of each input series, another hyperparameter you can choose; Number of input series is the number of explanatory variables you are using.