What is the best\correct data split approach over time-series data to compare performance of forecasting future data among ML and DL regressors?

Question

Let's say I have dataset contains a timestamp (non-standard timestamp column without datetime format) as a single feature and count as Label/target to predict within the following pandas dataframe format as follows:

   X        y
Timestamp label
+--------+-----+
|TS_24hrs|count|
+--------+-----+
|0       |157  |
|1       |334  |
|2       |176  |
|3       |86   |
|4       |89   |
 ...      ...
|270     |192  |
|271     |196  |
|270     |251  |
|273     |138  |
+--------+-----+
274 rows × 2 columns

I have already implemented RF regression within sklearn pipeline() after splitting data with the following strategy for 274 records:

• split data into [training-set + validation-set] Ref. e.g. The first 200 records [160 +40]
• keeping unseen [test-set] hold-on for final forecasting e.g. The last 74 records (after 200th rows\event)

#print(train.shape)          #(160, 2)
#print(validation.shape)     #(40, 2)
#print(test.shape)           #(74, 2)

Note: There are two further approaches especially used in DL regression models, which are not compatible with my split strategy and make the comparison of models' performance hard due to different shape size:

• lookback period,
• transform a time series dataset into a supervised learning dataset

Please see this post for further details.

I have tried 3 different data split approaches before fit(X,y) the regression model within pipeline as follows including\excluding y label:

# Load the time-series data as dataframe
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('/content/U2996_24hrs_.csv', sep=",")

# The first 200 records slice for training-set and validation-set
df200 = df[:200]          

# The rest records = 74 events (after 200th event) kept as hold-on unseen-set for forcasting
test = df[200:]                                   #test

# Split the data into training and testing sets
#---------------------Approach 1------------
from sklearn.model_selection import train_test_split
train, validation = train_test_split(df200 , test_size=0.2, shuffle=False) #train + validation
#---------------------------------------------

#---------------------Approach 2------------
from sklearn.model_selection import train_test_split
X = df200[['TS_24hrs']]
y = df200['count']
X_train, X_val, y_train, y_val = train_test_split(X, y , test_size=0.2, shuffle=False, random_state=0)
X_test = test['count'].values.reshape(-1,1)
#---------------------------------------------

#---------------------Approach 3------------
X = df200['TS_24hrs'][:160].values.reshape(-1,1)  #train
y = df200['count'][:160].values

Xv = df200['TS_24hrs'][160:].values.reshape(-1,1) #validation
yv = df200['count'][160:].values

Xt = test['TS_24hrs'].values.reshape(-1,1)        #test(unseen)
yt = test['count'].values
#---------------------------------------------

# Train and fit the RF model
from sklearn.ensemble import RandomForestRegressor
#rf_model = RandomForestRegressor(random_state=10).fit(train, train['count']) #X, y

# build an end-to-end pipeline, and supply the data into a regression model and train within the pipeline. It avoids leaking the test\val-set into the train-set
from sklearn.preprocessing import MinMaxScaler
from sklearn.ensemble import RandomForestRegressor
from sklearn.pipeline import Pipeline, make_pipeline

rf_pipeline1 = Pipeline([('scaler', MinMaxScaler()),('RF', RandomForestRegressor(random_state=10))]).fit(train, train['count']) #Approach 1 train-set includes label
rf_pipeline2 = Pipeline([('scaler', MinMaxScaler()),('RF', RandomForestRegressor(random_state=10))]).fit(X_train,y_train)  #Approach 2 train-set excludes label
rf_pipeline3 = Pipeline([('scaler', MinMaxScaler()),('RF', RandomForestRegressor(random_state=10))]).fit(X, y)     #Approach 3 train-set excludes label

# Displaying a Pipeline with a Preprocessing Step and Regression
from sklearn import set_config
set_config(display="text")
#print(rf_pipeline) 

# Use the pipeline to predict over the validation-set and test-set
y_predictions_test1  = rf_pipeline1.predict(test)
y_predictions_test2  = rf_pipeline2.predict(X_test)
y_predictions_test3  = rf_pipeline3.predict(Xt)

#y_predictions_val2  = rf_pipeline2.predict(X_val)
#y_predictions_val3  = rf_pipeline3.predict(Xv)

# Convert prediction result into dataframe for plot issue in ease
df_pre_test_rf1  = pd.DataFrame({'TS_24hrs':test['TS_24hrs'],        'count_prediction_test':y_predictions_test1})
df_pre_test_rf2  = pd.DataFrame({'TS_24hrs':test['TS_24hrs'],        'count_prediction_test':y_predictions_test2})
df_pre_test_rf3  = pd.DataFrame({'TS_24hrs':test['TS_24hrs'],        'count_prediction_test':y_predictions_test3})

#df_pre_val_rf2   = pd.DataFrame({'TS_24hrs':df200['TS_24hrs'][160:], 'count_prediction_val':y_predictions_val2})
#df_pre_val_rf3   = pd.DataFrame({'TS_24hrs':df200['TS_24hrs'][160:], 'count_prediction_val':y_predictions_val3})

# evaluate performance with MAE
# Evaluate performance by calculating the loss and metric over unseen test-set
from sklearn.metrics import mean_absolute_error, mean_squared_error, mean_absolute_percentage_error, explained_variance_score, r2_score 
rf_mae_test1  = mean_absolute_error(test['count'],                 df_pre_test_rf1['count_prediction_test'])
rf_mae_test2  = mean_absolute_error(test['count'],                 df_pre_test_rf2['count_prediction_test'])
rf_mae_test3  = mean_absolute_error(test['count'],                 df_pre_test_rf3['count_prediction_test'])

#rf_mae_val2   = mean_absolute_error(df200['count'][160:],          df_pre_val_rf2['count_prediction_val'])
#rf_mae_val3   = mean_absolute_error(df200['count'][160:],          df_pre_val_rf3['count_prediction_val'])

#visulize forecast or prediction of RF pipeline and TSS-based RF pipeline
import matplotlib.pyplot as plt
fig, ax = plt.subplots( figsize=(10,4))

train['count'].plot(label='Training-set', c='b')
validation['count'].plot(label='Validation-set', linestyle=':', c='b')
test['count'].plot(label='Test-set (unseen)', c='cyan')

#validation plot
#df_pre_val_rf2['count_prediction_val'].plot(label=f'RF_forecast_val (approach2)             MAE={rf_mae_val2:.2f}', linestyle=':', c='green', marker="2", alpha= 0.4)
#df_pre_val_rf3['count_prediction_val'].plot(label=f'RF_forecast_val (approach3)             MAE={rf_mae_val3:.2f}', linestyle=':', c='darkcyan',  marker="3", alpha= 0.4)

#predict plot
df_pre_test_rf1['count_prediction_test'].plot(label=f'RF_forecast_test (approach1)            MAE={rf_mae_test1:.2f}', linestyle='--', c='red',    marker="+")
df_pre_test_rf2['count_prediction_test'].plot(label=f'RF_forecast_test (approach2)            MAE={rf_mae_test2:.2f}', linestyle='--', c='green',    marker="+")
df_pre_test_rf3['count_prediction_test'].plot(label=f'RF_forecast_test (approach3)            MAE={rf_mae_test3:.2f}', linestyle='--', c='orange', marker="+", alpha= 0.4)


plt.legend()
plt.title('Plot of comparioson results of used implementation approaches trained RF pipeline ')
plt.ylabel('count', fontsize=15)
plt.xlabel('Timestamp [24hrs]', fontsize=15)
plt.show()

Got results like the following:

Output:

Q1: Which approach is correct? Why is the rest not correct?

1. Approach 1
2. Approach 2
3. Approach 3

Q2: Why does approach 2 has not acceptable prediction over its validation-set (it's like a constant line) while it has an acceptable forecast over an unseen test-set?

Q3: Do we need Hyper-parameters tuning to get optimum results by including it in the RF pipeline e.g, GridSearchCV()? I used but results didn't improve as follow:

# Load the time-series data as dataframe
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('/content/U2996_24hrs_.csv', sep=",")

# The first 200 records slice for training-set and validation-set
df200 = df[:200]          

# The rest records = 74 events (after 200th event) kept as hold-on unseen-set for forecasting
test = df[200:]                                   #test

# Split the data into training and testing sets
#---------------------Approach 2------------
from sklearn.model_selection import train_test_split
X = df200[['TS_24hrs']]
y = df200['count']
X_train, X_val, y_train, y_val = train_test_split(X, y , test_size=0.2, shuffle=False, random_state=0)
X_test = test['count'].values.reshape(-1,1)
#---------------------------------------------

# Train and fit the RF model
from sklearn.ensemble import RandomForestRegressor
#rf_model = RandomForestRegressor(random_state=10).fit(train, train['count']) #X, y

# build an end-to-end pipeline, and supply the data into a regression model and train within pipeline. It avoids leaking the test\val-set into the train-set
from sklearn.preprocessing import MinMaxScaler
from sklearn.ensemble import RandomForestRegressor
from sklearn.pipeline import Pipeline, make_pipeline

# Pipeline of Approach 2
rf_pipeline2  = Pipeline([('scaler', MinMaxScaler()),('RF', RandomForestRegressor(random_state=10))]).fit(X_train,y_train)   #Approach 2 train-set excludes label

# Pipeline of approach 2 (optimum)
# Parameters of pipelines can be set using '__' separated parameter names:
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import TimeSeriesSplit

tscv = TimeSeriesSplit(n_splits = 5)
param_grid = {
    "RF__n_estimators": [10, 50, 100],
    "RF__max_depth":    [1, 5, 10, 25],
    "RF__max_features": [*np.arange(0.1, 1.1, 0.1)],}

rf_pipeline2o = Pipeline([('scaler', MinMaxScaler()),('RF', GridSearchCV(rf_pipeline2,
                                                                         param_grid=param_grid,
                                                                         n_jobs=2,
                                                                         cv=tscv,
                                                                         refit=True))]).fit(X_train,y_train) #Approach 2 train-set excludes label

# Displaying a Pipeline with a Preprocessing Step and Regression
from sklearn import set_config
set_config(display="text")

#print(rf_pipeline2) 
#print(rf_pipeline2o)

# Use the pipeline to predict over the validation-set and test-set

y_predictions_test2  = rf_pipeline2.predict(X_test)
y_predictions_test2o = rf_pipeline2o.predict(X_test)

# Convert prediction result into dataframe for plot issue with ease
df_pre_test_rf2  = pd.DataFrame({'TS_24hrs':test['TS_24hrs'],        'count_prediction_test':y_predictions_test2})
df_pre_test_rf2o = pd.DataFrame({'TS_24hrs':test['TS_24hrs'],        'count_prediction_test':y_predictions_test2o})


# evaluate performance with MAE
# Evaluate performance by calculate the loss and metric over unseen test-set
from sklearn.metrics import mean_absolute_error, mean_squared_error, mean_absolute_percentage_error, explained_variance_score, r2_score 

rf_mae_test2  = mean_absolute_error(test['count'],                 df_pre_test_rf2['count_prediction_test'])
rf_mae_test2o = mean_absolute_error(test['count'],                 df_pre_test_rf2o['count_prediction_test'])

#visulize forecast or prediction of RF pipleine and TSS-based RF pipeline
import matplotlib.pyplot as plt
fig, ax = plt.subplots( figsize=(10,4))

train['count'].plot(label='Training-set', c='b')
validation['count'].plot(label='Validation-set', linestyle=':', c='b')
test['count'].plot(label='Test-set (unseen)', c='cyan')

#predict plot
df_pre_test_rf2['count_prediction_test'].plot(label=f'RF_forecast_test (approach2)              MAE={rf_mae_test2:.2f}', linestyle='--', c='green',    marker="+")
df_pre_test_rf2o['count_prediction_test'].plot(label=f'RF_forecast_test (approach2 opt.)       MAE={rf_mae_test2o:.2f}', linestyle='--', c='pink',    marker="+", alpha= 0.4)

plt.legend()
plt.title('Plot of comparioson results of used implementation approaches trained RF pipeline ')
plt.ylabel('count', fontsize=15)
plt.xlabel('Timestamp [24hrs]', fontsize=15)
plt.show()

---

Output: Sometimes slightly better\worsen than non-optimum pipeline\regression model despite setting random_state=10, but still partially predict constant value and I can't figure out why?

My observation so far shows: That is an issue with pipeline specification.

---

Edit: extend the experiment

Excluding the Timestamp column (feature X) and including it in the index within the dataframe

# Load the time-series data as dataframe
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('/content/U2996_24hrs_.csv', sep=",")
df = df.set_index('TS_24hrs')          #set index using timestamp <--------------------------------digest

# The first 200 records slice for training-set and validation-set
df200 = df[:200]          

# The rest records = 74 events (after 200th event) kept as hold-on unseen-set for forcasting
test = df[200:]  

# Split the data into training and testing sets
#---------------------Approach 1------------
from sklearn.model_selection import train_test_split
#X_train, X_test, y_train, y_test = train_test_split(X, y , test_size=0.27, shuffle=False, random_state=0)
train, validation = train_test_split(df200 , test_size=0.2, shuffle=False)
test = df[200:]           #the rest records (after 200th event) 
#---------------------------------------------

# ---------------------Approach 2------------
from sklearn.model_selection import train_test_split
X = df200.index.values.reshape(-1,1)
y = df200['count']
X_train, X_val, y_train, y_val = train_test_split(X, y , test_size=0.2, shuffle=False, random_state=0)
X_test = test['count'].values.reshape(-1,1)
# ---------------------------------------------

# ---------------------Approach 3------------
X = df200.index[:160].values.reshape(-1,1)  #train
y = df200['count'][:160].values

Xv = df200.index[160:].values.reshape(-1,1) #validation
yv = df200['count'][160:].values

Xt = test.index.values.reshape(-1,1)        #test(unseen)
yt = test['count'].values
# ---------------------------------------------

# Train and fit the RF model
from sklearn.ensemble import RandomForestRegressor
#rf_model = RandomForestRegressor().fit(train, train['count']) #X, y

# build an end-to-end pipeline, and supply the data into a regression model and train within pipeline. It avoids leaking the test\val-set into the train-set
from sklearn.preprocessing import MinMaxScaler
from sklearn.ensemble import RandomForestRegressor
from sklearn.pipeline import Pipeline, make_pipeline

rf_pipeline1 = Pipeline([('scaler', MinMaxScaler()),('RF', RandomForestRegressor(random_state=10))]).fit(train, train['count'])             #Approach 1 train-set includes label
rf_pipeline2 = Pipeline([('scaler', MinMaxScaler()),('RF', RandomForestRegressor(random_state=10))]).fit(X_train,y_train)                   #Approach 2 train-set excludes label
rf_pipeline3 = Pipeline([('scaler', MinMaxScaler()),('RF', RandomForestRegressor(random_state=10))]).fit(X, y)                              #Approach 3 train-set excludes label

# Displaying a Pipeline with a Preprocessing Step and Regression
from sklearn import set_config
set_config(display="text")
#print(rf_pipeline) 

# Use the pipeline to predict over the validation-set and test-set
y_predictions_test1  = rf_pipeline1.predict(test)
y_predictions_test2  = rf_pipeline2.predict(X_test)
y_predictions_test3  = rf_pipeline3.predict(Xt)

# Convert prediction result into dataframe for plot issue in ease
df_pre_test_rf1  = pd.DataFrame({'TS_24hrs':test.index,        'count_prediction_test':y_predictions_test1}).set_index('TS_24hrs')  
df_pre_test_rf2  = pd.DataFrame({'TS_24hrs':test.index,        'count_prediction_test':y_predictions_test2}).set_index('TS_24hrs')  
df_pre_test_rf3  = pd.DataFrame({'TS_24hrs':test.index,        'count_prediction_test':y_predictions_test3}).set_index('TS_24hrs') 

# evaluate performance with MAE
# Evaluate performance withby calculate the loss and metric over unseen test-set
from sklearn.metrics import mean_absolute_error, mean_squared_error, mean_absolute_percentage_error, explained_variance_score, r2_score 
rf_mae_test1  = mean_absolute_error(test['count'],                 df_pre_test_rf1['count_prediction_test'])
rf_mae_test2  = mean_absolute_error(test['count'],                 df_pre_test_rf2['count_prediction_test'])
rf_mae_test3  = mean_absolute_error(test['count'],                 df_pre_test_rf3['count_prediction_test'])

#visulize forecast or prediction of RF pipleine and TSS-based RF pipeline
import matplotlib.pyplot as plt
fig, ax = plt.subplots( figsize=(10,4))

train['count'].plot(label='Training-set', c='b')
validation['count'].plot(label='Validation-set', linestyle=':', c='b')
test['count'].plot(label='Test-set (unseen)', c='cyan')

#predict plot
df_pre_test_rf1['count_prediction_test'].plot(label=f'RF_forecast_test (approach1)            MAE={rf_mae_test1:.2f}', linestyle='--', c='red',    marker="+")
df_pre_test_rf2['count_prediction_test'].plot(label=f'RF_forecast_test (approach2)            MAE={rf_mae_test2:.2f}', linestyle='--', c='green',    marker="+")
df_pre_test_rf3['count_prediction_test'].plot(label=f'RF_forecast_test (approach3)            MAE={rf_mae_test3:.2f}', linestyle='--', c='orange', marker="+", alpha= 0.4)

plt.legend()
plt.title('The plot of comparison results of used implementation approaches trained RF pipeline \n by excluding the Timestamp column (feature X) and including it in the index within the dataframe')
plt.ylabel('count', fontsize=15)
plt.xlabel('Timestamp [24hrs]', fontsize=15)
plt.show()

Dropping timestamp and considering target column as both feature X and label y (duplication)

# Load the time-series data as dataframe
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('/content/U2996_24hrs_.csv', sep=",")
df = df.set_index('TS_24hrs')          #set index using timestamp <--------------------------------digest

# The first 200 records slice for training-set and validation-set
df200 = df[:200]          

# The rest records = 74 events (after 200th event) kept as hold-on unseen-set for forcasting
test = df[200:]  

# Split the data into training and testing sets
#---------------------Approach 1------------
from sklearn.model_selection import train_test_split
#X_train, X_test, y_train, y_test = train_test_split(X, y , test_size=0.27, shuffle=False, random_state=0)
train, validation = train_test_split(df200 , test_size=0.2, shuffle=False)
test = df[200:]           #the rest records (after 200th event) 
#---------------------------------------------

# ---------------------Approach 2------------
from sklearn.model_selection import train_test_split
X = df200.index.values.reshape(-1,1)
y = df200['count']
X_train, X_val, y_train, y_val = train_test_split(X, y , test_size=0.2, shuffle=False, random_state=0)
X_test = test['count'].values.reshape(-1,1)
# ---------------------------------------------

# ---------------------Approach 3------------
X = df200.index[:160].values.reshape(-1,1)  #train
y = df200['count'][:160].values

Xv = df200.index[160:].values.reshape(-1,1) #validation
yv = df200['count'][160:].values

Xt = test.index.values.reshape(-1,1)        #test(unseen)
yt = test['count'].values
# ---------------------------------------------

# Train and fit the RF model
from sklearn.ensemble import RandomForestRegressor
#rf_model = RandomForestRegressor().fit(train, train['count']) #X, y

# build an end-to-end pipeline, and supply the data into a regression model and train within pipeline. It avoids leaking the test\val-set into the train-set
from sklearn.preprocessing import MinMaxScaler
from sklearn.ensemble import RandomForestRegressor
from sklearn.pipeline import Pipeline, make_pipeline

rf_pipeline1 = Pipeline([('scaler', MinMaxScaler()),('RF', RandomForestRegressor(random_state=10))]).fit(train, train['count'])             #Approach 1 train-set includes label
rf_pipeline2 = Pipeline([('scaler', MinMaxScaler()),('RF', RandomForestRegressor(random_state=10))]).fit(X_train,y_train)                   #Approach 2 train-set excludes label
rf_pipeline3 = Pipeline([('scaler', MinMaxScaler()),('RF', RandomForestRegressor(random_state=10))]).fit(X, y)                              #Approach 3 train-set excludes label

# Displaying a Pipeline with a Preprocessing Step and Regression
from sklearn import set_config
set_config(display="text")
#print(rf_pipeline) 

# Use the pipeline to predict over the validation-set and test-set
y_predictions_test1  = rf_pipeline1.predict(test)
y_predictions_test2  = rf_pipeline2.predict(X_test)
y_predictions_test3  = rf_pipeline3.predict(Xt)

# Convert prediction result into dataframe for plot issue in ease
df_pre_test_rf1  = pd.DataFrame({'TS_24hrs':test.index,        'count_prediction_test':y_predictions_test1}).set_index('TS_24hrs')  
df_pre_test_rf2  = pd.DataFrame({'TS_24hrs':test.index,        'count_prediction_test':y_predictions_test2}).set_index('TS_24hrs')  
df_pre_test_rf3  = pd.DataFrame({'TS_24hrs':test.index,        'count_prediction_test':y_predictions_test3}).set_index('TS_24hrs') 

# evaluate performance with MAE
# Evaluate performance withby calculate the loss and metric over unseen test-set
from sklearn.metrics import mean_absolute_error, mean_squared_error, mean_absolute_percentage_error, explained_variance_score, r2_score 
rf_mae_test1  = mean_absolute_error(test['count'],                 df_pre_test_rf1['count_prediction_test'])
rf_mae_test2  = mean_absolute_error(test['count'],                 df_pre_test_rf2['count_prediction_test'])
rf_mae_test3  = mean_absolute_error(test['count'],                 df_pre_test_rf3['count_prediction_test'])

#visulize forecast or prediction of RF pipleine and TSS-based RF pipeline
import matplotlib.pyplot as plt
fig, ax = plt.subplots( figsize=(10,4))

train['count'].plot(label='Training-set', c='b')
validation['count'].plot(label='Validation-set', linestyle=':', c='b')
test['count'].plot(label='Test-set (unseen)', c='cyan')

#predict plot
df_pre_test_rf1['count_prediction_test'].plot(label=f'RF_forecast_test (approach1)            MAE={rf_mae_test1:.2f}', linestyle='--', c='red',    marker="+")
df_pre_test_rf2['count_prediction_test'].plot(label=f'RF_forecast_test (approach2)            MAE={rf_mae_test2:.2f}', linestyle='--', c='green',    marker="+")
df_pre_test_rf3['count_prediction_test'].plot(label=f'RF_forecast_test (approach3)            MAE={rf_mae_test3:.2f}', linestyle='--', c='orange', marker="+", alpha= 0.4)

plt.legend()
plt.title('The plot of comparison results of used implementation approaches trained RF pipeline \n by dropping timestamp and considering target column as both feature X and label y (duplication)')
plt.ylabel('count', fontsize=15)
plt.xlabel('Timestamp [24hrs]', fontsize=15)
plt.show()

Q4:

• Is it a must not to use the timestamp column in the frame and integrate it into the index ? If yes, Should we drop Timestamp: TS_24hrs [X] and keep count [y] as the label only in the frame?
• Duplication of Timestamp: TS_24hrs [X] after ingesting it into index and showing to model as the feature, is it matter?

My observation through this experiment shows sklearn ignores indices considering information passing to .fit(). It seems including\excluding timestamp column in dataframe has no impact on all approaches and it doesn't matter whether:

• that is through having two columns (Timestamp: TS_24hrs [X] and target column count [y]) in a dataframe
• or using the index/values in a dataframe. !

Is there any explanation for this?

Answer 1

One possible issue with inconsistent behavior might be because scikit-learn is designed to learn to weight features to predict the targets. The Timestamp column is an index, not a feature according to scikit-learn. The dataset does not have any features according to scikit-learn. The count column is the target.

A better option would be to use a package specifically designed for time series data.