# Why is performance worse when my time-series data is not shuffled prior to a train/test split vs. when it is shuffled prior to the split?

We are running RandomForest model on a time-series data. The model is run in real time and is refit every time a new row is added. Since it is a timeseries data, we set shuffle to false while splitting into train and test dataset.

We observed that there is a drastic change in scores when shuffle is True and when shuffle is false

The code being used is as follows

# Set shuffle = 'True' or 'False'
X = df.drop(columns=['label'])
y = df['label']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.05, shuffle=True)

count = 0
predictions = []

for idx in X_test.index.values:
# Train the model on training data
# print(count, X_train.shape, y_train.shape)
rf = RandomForestRegressor(n_estimators = 600, max_depth = 7, random_state = 12345)
rf.fit(X_train, y_train)

predictions.append(rf.predict(X_test.loc[X_test.index == idx]))
# print(len(predictions))

X_train.loc[len(X_train)] = X_test.loc[idx]
y_train.loc[len(y_train)] = y_test.loc[idx]
count+=1


Initially, we thought the difference is due to covariance shift in the data. But that shouldn't affect this much for continuous fit

Here are screenshots for the prediction plot

With shuffle = False

With shuffle = True

• Can you show a plot of the entire data set in both cases (including the training set, not just the test set)?
– Wes
Feb 18, 2019 at 0:39
• Also, can you give the actual performance of the models, and not just the predictions (i.e., a numerical value like MSE, etc.)?
– Wes
Feb 18, 2019 at 0:42
• When shuffle = True, 'mae': 0.012749809403589319, 'r2score':0.534131151271332705, 'rmse': 0.01478679726017944. When shuffle = False, 'mae': 0.012631170478535453, 'r2score': -0.03146366881412077, 'rmse': 0.020236256497426223 Links for training set plots, shuffle = False : i.imgur.com/GYAQup9.png, shuffle = True : i.imgur.com/b9cATse.png Feb 18, 2019 at 7:28
• What is happening when the target variable is 0? Is this a valid result? You have a short section of 0 all in a row before it is shuffled.
– Wes
Feb 18, 2019 at 14:48
• Also, it is probably useful for you to look at histograms of your features and target variables in the training set vs. the test set in both cases of not shuffling and shuffling.
– Wes
Feb 18, 2019 at 14:57