0
$\begingroup$

I have a 2-D dataframe df:

               Close       Low      High  ...    usdchf    usdjpy    Target
2008-01-15 -0.004499 -0.001492  0.000537  ... -0.001464 -0.013672 -0.009326
2008-01-16 -0.009326 -0.011368 -0.004086  ...  0.006964  0.005385 -0.001640
2008-01-17 -0.001640 -0.001022 -0.010367  ...  0.001456 -0.008803 -0.001505
2008-01-18 -0.001505  0.000409 -0.001028  ... -0.001090  0.004041  0.003283
2008-01-21  0.003283 -0.012530 -0.006118  ...  0.009915 -0.011326 -0.001228
...              ...       ...       ...  ...       ...       ...       ...
2021-01-25  0.000260 -0.002860 -0.000164  ... -0.000042  0.002618 -0.001907
2021-01-26 -0.001907 -0.000727 -0.000903  ...  0.003046 -0.000164  0.001379
2021-01-27  0.001379 -0.003835 -0.000298  ... -0.001486 -0.001513 -0.004762
2021-01-28 -0.004762  0.001691 -0.002249  ...  0.002481  0.005212  0.001237
2021-01-29  0.001237  0.001125  0.001071  ... -0.000327  0.001421  0.000315

[3404 rows x 11 columns]

df["Target"] = df.Close.shift(-1)

The target variable y is the column "Target" which is the "Close" column shifted 1 day to make 1-day forward returns (as shown above), and the rest are the predictor features. So, essentially I'm looking to predict one-day ahead. My question is what is the best approach to prepare this data for use in a LSTM model?

I understand that the input shape for my sequential LSTM model is in the form (samples, time steps (mini-batches), features). So, features would be 10, I can have say 50 as time steps and samples will be 80% of dataframe length.

I found this code online:

# Convert the data to numpy values
np_data_unscaled = np.array(data_filtered)
np_data = np.reshape(np_data_unscaled, (nrows, -1))
print(np_data.shape)

# Transform the data by scaling each feature to a range between 0 and 1
scaler = MinMaxScaler()
np_data_scaled = scaler.fit_transform(np_data_unscaled)

# Creating a separate scaler that works on a single column for scaling predictions
scaler_pred = MinMaxScaler()
df_Close = pd.DataFrame(data_filtered_ext['Close'])
np_Close_scaled = scaler_pred.fit_transform(df_Close)


# Set the sequence length - this is the timeframe used to make a single prediction
sequence_length = 50

# Split the training data into train and train data sets
# As a first step, we get the number of rows to train the model on 80% of the data 
train_data_len = math.ceil(np_data_scaled.shape[0] * 0.8)

# Create the training and test data
train_data = np_data_scaled[0:train_data_len, :]
test_data = np_data_scaled[train_data_len - sequence_length:, :]

# The RNN needs data with the format of [samples, time steps, features]
# Here, we create N samples, sequence_length time steps per sample, and 6 features
def partition_dataset(sequence_length, data):
    x, y = [], []
    data_len = data.shape[0]
    for i in range(sequence_length, data_len):
        x.append(data[i-sequence_length:i,:]) #contains sequence_length values 0-sequence_length * columsn
        y.append(data[i, index_Close]) #contains the prediction values for validation,  for single-step prediction
    
    # Convert the x and y to numpy arrays
    x = np.array(x)
    y = np.array(y)
    return x, y

# Generate training data and test data
x_train, y_train = partition_dataset(sequence_length, train_data)
x_test, y_test = partition_dataset(sequence_length, test_data)

# Print the shapes: the result is: (rows, training_sequence, features) (prediction value, )
print(x_train.shape, y_train.shape)
print(x_test.shape, y_test.shape)

# Validate that the prediction value and the input match up
# The last close price of the second input sample should equal the first prediction value
print(x_train[1][sequence_length-1][index_Close])
print(y_train[0])

The thing I want to ensure is that the data is cross-validated whilst respecting temporal dependencies and that minmax is applied on the partitions to prevent data leakage. How can this be done?

Also, when they separate with test_data = np_data_scaled[train_data_len - sequence_length:, :],is this data peeking? I understand that the def partition_dataset() uses a sliding window approach and so I am confused as to whether this is okay?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.