2
$\begingroup$

We are running RandomForest model on a time-series data. The model is run in real time and is refit every time a new row is added. Since it is a timeseries data, we set shuffle to false while splitting into train and test dataset.

We observed that there is a drastic change in scores when shuffle is True and when shuffle is false

The code being used is as follows

# Set shuffle = 'True' or 'False'
df = pandas.read_csv('data.csv', index_col=0)
X = df.drop(columns=['label'])
y = df['label']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.05, shuffle=True)

count = 0 
predictions = [] 

for idx in X_test.index.values:
    # Train the model on training data
    # print(count, X_train.shape, y_train.shape)
    rf = RandomForestRegressor(n_estimators = 600, max_depth = 7, random_state = 12345)
    rf.fit(X_train, y_train)

    predictions.append(rf.predict(X_test.loc[X_test.index == idx]))
    # print(len(predictions))

    X_train.loc[len(X_train)] = X_test.loc[idx]
    y_train.loc[len(y_train)] = y_test.loc[idx]
    count+=1

Initially, we thought the difference is due to covariance shift in the data. But that shouldn't affect this much for continuous fit

Here are screenshots for the prediction plot

With shuffle = False enter image description here

With shuffle = True enter image description here

$\endgroup$
7
  • $\begingroup$ Can you show a plot of the entire data set in both cases (including the training set, not just the test set)? $\endgroup$
    – Wes
    Feb 18, 2019 at 0:39
  • $\begingroup$ Also, can you give the actual performance of the models, and not just the predictions (i.e., a numerical value like MSE, etc.)? $\endgroup$
    – Wes
    Feb 18, 2019 at 0:42
  • $\begingroup$ When shuffle = True, 'mae': 0.012749809403589319, 'r2score':0.534131151271332705, 'rmse': 0.01478679726017944. When shuffle = False, 'mae': 0.012631170478535453, 'r2score': -0.03146366881412077, 'rmse': 0.020236256497426223 Links for training set plots, shuffle = False : i.imgur.com/GYAQup9.png, shuffle = True : i.imgur.com/b9cATse.png $\endgroup$ Feb 18, 2019 at 7:28
  • $\begingroup$ What is happening when the target variable is 0? Is this a valid result? You have a short section of 0 all in a row before it is shuffled. $\endgroup$
    – Wes
    Feb 18, 2019 at 14:48
  • $\begingroup$ Also, it is probably useful for you to look at histograms of your features and target variables in the training set vs. the test set in both cases of not shuffling and shuffling. $\endgroup$
    – Wes
    Feb 18, 2019 at 14:57

2 Answers 2

2
$\begingroup$

Without looking further into the data myself, I can surmise that something has changed recently with your data such that if you split without shuffling, some aspect of the data in your test set (which is what you most recently collected) is underrepresented in your training set. By shuffling the data, you allow those more recent samples to also be present in your training set, and thus your test set performance improves.

$\endgroup$
0
$\begingroup$

Shuffling should be false in time series models because otherwise, you will be training the model on patterns it does not yet have access to.

At each timestep, the model should only be trained up to the point of data visibility. e.g. at timestep 10, model should only be trained with data from 0 to 10 without visibality of data from 11 to 40.

Otherwise you will get great model results now but when you implement the model today (2021), you get poor actual performance because you dont get to train the model with future 2023 patterns/data.

$\endgroup$
1
  • $\begingroup$ What would be the way to keep the model performance with time? $\endgroup$ Nov 27, 2021 at 9:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.