Proper way to make Train/test split on Time-Series

Question

I want to create a model with LSTM to predict a user the next purchase value. For this I have used I used a user's purchase history. I have created the model and it works well, but honestly, I don't know I do the Train/Test split on the proper way or not.

To do this, I have used (univariate) user's purchase history.(X-purchase history values, y-target purchase value) As a first step, I have created a sliding-window process that creates new data. (As you can see in the pic) In the original dataset, I had 1000 users with 2820 timestamps and 1 feature (purchase values), with the Sliding-window process I got 1000*2320 users with 500 timestamps and 1 feature.

X.shape -> OriginalDataShape (1000, 2820, 1)

X.shape -> ModifiedDataShape (2320000‬, 500, 1)

# model
model = Sequential()
model.add(LSTM(50, activation='relu', input_shape=(500,1))) model.add(Dense(1)) model.compile(loss='mae', optimizer='adam')


# train
repeats = range(3)
scores = list()
for i in repeats:
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
    model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=5, batch_size=32)
    pred = model.predict(X_test)
    score.append(metrics.mean_squared_error(pred,y_test))
print('Final score (Mse):')
print(score)

My questions are: It is the proper way or not? If it is not, do you have a suggestion or GitHub link for the solution?

Answer 1

The problem here is that you're shuffling the time-series before splitting it.
This way, every time-step in the test set might have a time-step close to it in the train set.

To avoid this, you can set shuffle=False in train_test_split (so that the train set is before the test set), or use Group K-Fold with the date as the group (so whole days are either in the train or test set).

You can read more in this question in Cross Validated