# Regression performance varies hugely on shuffling training and testing data

I'm working on a regression problem to predict a variable y based on an input vector X with about 10 columns. To split the data for training and testing, I use the test_train_split method:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle = True)


Here, it can be seen that X and y are shuffled entirely along with their indices. For the purpose of this example, I will consider only one algorithm, RandomForestRegressor:

regr1 = RandomForestRegressor(n_estimators = 200)
regr1.fit(X_train, y_train)
y_pred = regr1.predict(X_test)


When I train the regressor with X_train and y_train from the above method, I get certain results which I consider good. For my application, I would need the model to predict for cases with sequential data (as they would be if not shuffled).

Therefore, I tried the following method to shuffle the training data alone and keep the testing data as is.

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle = False)
train_data = pd.concat([X_train, y_train], axis = 1)
train_data = train_data.sample(frac=1)
X_train = train_data.iloc[:,0:-1]
y_train = train_data.iloc[:,-1]


In the above method, I first split the data into test/train indices and then shuffle the training data alone and keep the testing data with their original indices. When I train the regressor with the exact same parameters as before and test with X_test, I get significantly poorer results.

I have also tried with both train and test data without shuffling and got bad results as well. I want to be able to train the model so that it can predict for unknown values coming in a sequential order (as it would be in real time).

I'm not able to understand why the shuffling of the test data alone affects the performance, as the model should merely be predicting based on the trained parameters which entirely depend on the training data.

• Does this answer your question? – ncasas Sep 28 at 8:06
• @ncasas Thanks for your reply. It's really strange because the performance is poorer when I don't shuffle the data before training the RF than when I do. This is also the same for other models. As I have mentioned, the results are strongly affected by how the dataset is split. If I split the training and testing set by setting the parameter shuffle = True, I get a better result as compared to shuffle = False. – prame Sep 28 at 15:02