Input Data for LSTM RNN transformation

Question

I am new to RNN, and I do not know how to transform my input data in order to use it in a LSTM layer.

Currently, my input data has (1000 samples, 50 features). But let´s say it is only (1000 samples, 2 features) to make the example below easier. Each sample would look like this:

[Date, Score, team_id]
1.[11/10/2022, 56, 23543]
2.[12/10/2022, 75, 44354]
3.[17/10/2022, 56, 44354]....

I would like to use 5 time steps, so my data will have a shape like this (1000, 5, 50). I know I would need to get the 5 previous scores for all samples, but once I do that, I do not understand how to represent the data. I guess it will be like a list of matrices, but how is that coded from the dataframe I have?

UPDATE! I have created the list of matrices where if you print(list[0]), you get an array of dicts with the information, so we could say it is a list of lists of dicts. How do I convert that into a dataframe or input data?

Thank you very much

Answer 1

You can use the available function Keras or any other library though not very commonly available.
This will give you a lot of functionalities out of the box. tf link

import pandas as pd
data=pd.read_csv("/content/sample_data/california_housing_test.csv")
from tensorflow.keras.preprocessing import timeseries_dataset_from_array

seq_len=5
tf_x = timeseries_dataset_from_array(
    data, None, sequence_length=seq_len, batch_size=32, end_index=data.shape[0]-seq_len )
for x in tf_x.take(1):
    print(x.shape)
    print("Data - ",x[0,0])

You can create your own using Pandas shift and Numpy concatenate.
Create n(seq_len) array of the initial data frame by shifting. Then concatenate all of them together on 1st axis.

import numpy as np
seq_len=5

list_np_arr=[]
for i in range(seq_len):
    list_np_arr.append(data.shift(i+1).to_numpy()[:,np.newaxis,:])
# shape = # df_size-2*seq_len, seq_len, n_features
np.concatenate((list_np_arr), axis=1)[seq_len:-seq_len,:,:].shape 

Slicing (i.e. seq_len:-seq_len) was done to remove the initial and ending portion as it will have NaN of the shifted data.