We are running RandomForest model on a time-series data. The model is run in real time and is refit every time a new row is added. Since it is a timeseries data, we set shuffle to false while splitting into train and test dataset.
We observed that there is a drastic change in scores when shuffle is True and when shuffle is false
The code being used is as follows
# Set shuffle = 'True' or 'False' df = pandas.read_csv('data.csv', index_col=0) X = df.drop(columns=['label']) y = df['label'] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.05, shuffle=True) count = 0 predictions =  for idx in X_test.index.values: # Train the model on training data # print(count, X_train.shape, y_train.shape) rf = RandomForestRegressor(n_estimators = 600, max_depth = 7, random_state = 12345) rf.fit(X_train, y_train) predictions.append(rf.predict(X_test.loc[X_test.index == idx])) # print(len(predictions)) X_train.loc[len(X_train)] = X_test.loc[idx] y_train.loc[len(y_train)] = y_test.loc[idx] count+=1
Initially, we thought the difference is due to covariance shift in the data. But that shouldn't affect this much for continuous fit
Here are screenshots for the prediction plot