Keras LSTM Dimensions

Question

I am trying to build a RNN in Keras. I am inputting an array of 300K values. I have 4 independent variables (W,X,Y,Z) And 1 dependent variable f(W,X,Y,Z).

The array is then split 270K for training and 30K for validation.

When I try to put my data into the network, it says "expected ndim=3, found ndim=4".

My model looks like this

model=Sequential()
model.add(LSTM(4, input_shape=(270000,4)))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer ='adam')
model.fit(X_tr,Y_tr, validation_data=(X_val, Y_val),
          epochs=10, batch_size=30000, verbose=1)

Answer 1

When creating an RNN, we generally assume there is some temporal correlation, for example: that the data has a time-series nature, like the price of a stock. We might consider a window of 30 days as the timestep for a single sample.

When creating input data for an LSTM layer, you need to consider how many timesteps are included in one of your samples, and put the data into that shape. The input shape should be (number_of_sample, num_timesteps, num_features):

Take a look at this answer for more information.

Your data then might be able to be reshaped:

new_shape = data.reshape((None, 100, 4))       # new input shape to LSTM

• None means you let the number of samples be whatever it must in order to fit your original data shape.
• 100 means you pass 100 sequential datapoints/timesteps as a single sample
• 4 is your number of independent variables (features)

That reshape might throw an error if your data cannot be reshaped to make the dimensions you want. You might need to trim your data (i.e. leave some of teh oldest samples out)