0
$\begingroup$

I am new to RNN, and I do not know how to transform my input data in order to use it in a LSTM layer.

Currently, my input data has (1000 samples, 50 features). But let´s say it is only (1000 samples, 2 features) to make the example below easier. Each sample would look like this:

[Date, Score, team_id]
1.[11/10/2022, 56, 23543]
2.[12/10/2022, 75, 44354]
3.[17/10/2022, 56, 44354]....

I would like to use 5 time steps, so my data will have a shape like this (1000, 5, 50). I know I would need to get the 5 previous scores for all samples, but once I do that, I do not understand how to represent the data. I guess it will be like a list of matrices, but how is that coded from the dataframe I have?

UPDATE! I have created the list of matrices where if you print(list[0]), you get an array of dicts with the information, so we could say it is a list of lists of dicts. How do I convert that into a dataframe or input data?

Thank you very much

$\endgroup$

1 Answer 1

1
$\begingroup$

You can use the available function Keras or any other library though not very commonly available.
This will give you a lot of functionalities out of the box. tf link

import pandas as pd
data=pd.read_csv("/content/sample_data/california_housing_test.csv")
from tensorflow.keras.preprocessing import timeseries_dataset_from_array

seq_len=5
tf_x = timeseries_dataset_from_array(
    data, None, sequence_length=seq_len, batch_size=32, end_index=data.shape[0]-seq_len )
for x in tf_x.take(1):
    print(x.shape)
    print("Data - ",x[0,0])

You can create your own using Pandas shift and Numpy concatenate.
Create n(seq_len) array of the initial data frame by shifting. Then concatenate all of them together on 1st axis.

import numpy as np
seq_len=5

list_np_arr=[]
for i in range(seq_len):
    list_np_arr.append(data.shift(i+1).to_numpy()[:,np.newaxis,:])
# shape = # df_size-2*seq_len, seq_len, n_features
np.concatenate((list_np_arr), axis=1)[seq_len:-seq_len,:,:].shape 

Slicing (i.e. seq_len:-seq_len) was done to remove the initial and ending portion as it will have NaN of the shifted data.

$\endgroup$
1
  • $\begingroup$ Hello, thanks for your answer! What type of variable 'data' is in your code? Because in my program it is a list, and it does not support some of the methods like shape or shift... $\endgroup$ Oct 26, 2022 at 8:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.