5
$\begingroup$

Normally stacking algorithm uses K-fold cross validation technique to predict oof validation that used for level 2 prediction.

In case of time-series data (say stock movement prediction), K-fold cross validation can't be used and time-series validation (one suggested on sklearn lib) is suitable to evaluate the model performance. In this case no prediction shall be made on first fold and no training shall be made on last fold. How do we use stacking algorithm cross validation technique for time-series data?

$\endgroup$

2 Answers 2

5
$\begingroup$

TL;DR

Time-series algorithms assume that data points are ordered. Traditional K-Fold cannot be used for time series because it doesn't take into account the order in which data points appear. One approach to validate time series algorithms is with Time Based Splitting.

K-Fold vs Time Based Splitting

The two graphs below show the difference between K-Fold and Time Based Splitting. From them, the following characteristics can be observed.


K-Fold always the all data points.

Time Base Splitting uses a fraction of all data points.


K-Fold lets the test set be any data point.

Time Base Splitting only allows the test set to have higher indexes than the training set.


K-Fold will use the first data point for testing and the last data point for training.

Time Base Splitting will never use the first data point for testing and never use the last data point for training.

enter image description here TimeSeriesSplit plot

Scikit-learn implementation

Scikit-learn has an implementation of this algorithm called TimeSeriesSplit.

Look at their documentation, you find the following example:

from sklearn.model_selection import TimeSeriesSplit
X = np.array([[1, 2], [3, 4], [1, 2], [3, 4], [1, 2], [3, 4]])
y = np.array([1, 2, 3, 4, 5, 6])
tscv = TimeSeriesSplit(n_splits=5)

for train_index, test_index in tscv.split(X):
   print("TRAIN:", train_index, "TEST:", test_index)
   X_train, X_test = X[train_index], X[test_index]
   y_train, y_test = y[train_index], y[test_index]

>> TRAIN: [0] TEST: [1]
>> TRAIN: [0 1] TEST: [2]
>> TRAIN: [0 1 2] TEST: [3]
>> TRAIN: [0 1 2 3] TEST: [4]
>> TRAIN: [0 1 2 3 4] TEST: [5]
$\endgroup$
3
  • 2
    $\begingroup$ Thanks. When we ensemble base models (especially stacking) usually out of fold predictions of different base models are used to train meta model to avoid over fitting. In Time series cross validation, first data point will never be used for testing and last data point will never be used for training. Hence, we loose out of fold predictions for first data point and fitting last point... i.e., In coding example, we have folds [0,1,2,3,4,5] and we get out of fold predictions for [1,2,3,4,5]. Is that fine? $\endgroup$ Commented Nov 19, 2018 at 3:15
  • $\begingroup$ @ManimaranSubramanian that is fine. You are right to say that you cannot predict for fold 0. $\endgroup$ Commented Nov 20, 2018 at 21:19
  • $\begingroup$ I dont think even time series split cv is appropriate for time series prediction because it trains on those 4 blocks which capture very unique stretches of the variable's movement, in other words 4 market regimes, which are unlikely to re-appear out-of-sample. I think more randomization is needed while still retaining the order of time events. Cant bootstrap resampling with replacement be used within stacking? $\endgroup$
    – develarist
    Commented Apr 14, 2020 at 22:30
3
$\begingroup$

Standard TimeSeriesSplit from sklearn is not able to work with StackingRegressor because StackingRegressor uses cross_val_predict under the hood. This will result in errors like:

cross_val_predict only works for partitions

To make a stacking with time-series and sklearn models you simply have to write these few lines of code...

from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import TimeSeriesSplit

# generate dummy data
X = np.random.uniform(0,1, (1000,10))
y = np.random.uniform(0,1, (1000,))

# initialize two models to be stacked
rf = RandomForestRegressor()
gb = GradientBoostingRegressor()

# generate cross-val-prediction with rf and gb using TimeSeriesSplit
cross_val_predict = np.row_stack([
    np.column_stack([
        rf.fit(X[id_train], y[id_train]).predict(X[id_test]),
        gb.fit(X[id_train], y[id_train]).predict(X[id_test]),
        y[id_test]  # we add in the last position the corresponding fold labels
    ])
    for id_train,id_test in TimeSeriesSplit(n_splits=3).split(X)
])  # (test_size*n_splits, n_models_to_stack+1)

# final fit rf and gb with all the available data
rf.fit(X,y)
gb.fit(X,y)

# fit a linear stacking on cross_val_predict
stacking = LinearRegression()
stacking.fit(cross_val_predict[:,:-1], cross_val_predict[:,-1])

# how generate predictions on new unseen data
X_new = np.random.uniform(0,1, (30,10))
pred = stacking.predict(
    np.column_stack([
        rf.predict(X_new),
        gb.predict(X_new)
    ])
)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.