2
$\begingroup$

We are running RandomForest model on a time-series data. The model is run in real time and is refit every time a new row is added. Since it is a timeseries data, we set shuffle to false while splitting into train and test dataset.

We observed that there is a drastic change in scores when shuffle is True and when shuffle is false

The code being used is as follows

# Set shuffle = 'True' or 'False'
df = pandas.read_csv('data.csv', index_col=0)
X = df.drop(columns=['label'])
y = df['label']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.05, shuffle=True)

count = 0 
predictions = [] 

for idx in X_test.index.values:
    # Train the model on training data
    # print(count, X_train.shape, y_train.shape)
    rf = RandomForestRegressor(n_estimators = 600, max_depth = 7, random_state = 12345)
    rf.fit(X_train, y_train)

    predictions.append(rf.predict(X_test.loc[X_test.index == idx]))
    # print(len(predictions))

    X_train.loc[len(X_train)] = X_test.loc[idx]
    y_train.loc[len(y_train)] = y_test.loc[idx]
    count+=1

Initially, we thought the difference is due to covariance shift in the data. But that shouldn't affect this much for continuous fit

Here are screenshots for the prediction plot

With shuffle = False enter image description here

With shuffle = True enter image description here

$\endgroup$
7
  • $\begingroup$ Can you show a plot of the entire data set in both cases (including the training set, not just the test set)? $\endgroup$ – Wes Feb 18 '19 at 0:39
  • $\begingroup$ Also, can you give the actual performance of the models, and not just the predictions (i.e., a numerical value like MSE, etc.)? $\endgroup$ – Wes Feb 18 '19 at 0:42
  • $\begingroup$ When shuffle = True, 'mae': 0.012749809403589319, 'r2score':0.534131151271332705, 'rmse': 0.01478679726017944. When shuffle = False, 'mae': 0.012631170478535453, 'r2score': -0.03146366881412077, 'rmse': 0.020236256497426223 Links for training set plots, shuffle = False : i.imgur.com/GYAQup9.png, shuffle = True : i.imgur.com/b9cATse.png $\endgroup$ – Sumesh Surendran Feb 18 '19 at 7:28
  • $\begingroup$ What is happening when the target variable is 0? Is this a valid result? You have a short section of 0 all in a row before it is shuffled. $\endgroup$ – Wes Feb 18 '19 at 14:48
  • $\begingroup$ Also, it is probably useful for you to look at histograms of your features and target variables in the training set vs. the test set in both cases of not shuffling and shuffling. $\endgroup$ – Wes Feb 18 '19 at 14:57
2
$\begingroup$

Without looking further into the data myself, I can surmise that something has changed recently with your data such that if you split without shuffling, some aspect of the data in your test set (which is what you most recently collected) is underrepresented in your training set. By shuffling the data, you allow those more recent samples to also be present in your training set, and thus your test set performance improves.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.