Normally stacking algorithm uses K-fold cross validation technique to predict oof validation that used for level 2 prediction.

In case of time-series data (say stock movement prediction), K-fold cross validation can't be used and time-series validation (one suggested on sklearn lib) is suitable to evaluate the model performance. In this case no prediction shall be made on first fold and no training shall be made on last fold. How do we use stacking algorithm cross validation technique for time-series data?



Time-series algorithms assume that data points are ordered. Traditional K-Fold cannot be used for time series because it doesn't take into account the order in which data points appear. One approach to validate time series algorithms is with Time Based Splitting.

K-Fold vs Time Based Splitting

The two graphs below show the difference between K-Fold and Time Based Splitting. From them, the following characteristics can be observed.

K-Fold always the all data points.

Time Base Splitting uses a fraction of all data points.

K-Fold lets the test set be any data point.

Time Base Splitting only allows the test set to have higher indexes than the training set.

K-Fold will use the first data point for testing and the last data point for training.

Time Base Splitting will never use the first data point for testing and never use the last data point for training.

enter image description here TimeSeriesSplit plot

Scikit-learn implementation

Scikit-learn has an implementation of this algorithm called TimeSeriesSplit.

Look at their documentation, you find the following example:

from sklearn.model_selection import TimeSeriesSplit
X = np.array([[1, 2], [3, 4], [1, 2], [3, 4], [1, 2], [3, 4]])
y = np.array([1, 2, 3, 4, 5, 6])
tscv = TimeSeriesSplit(n_splits=5)

for train_index, test_index in tscv.split(X):
   print("TRAIN:", train_index, "TEST:", test_index)
   X_train, X_test = X[train_index], X[test_index]
   y_train, y_test = y[train_index], y[test_index]

>> TRAIN: [0] TEST: [1]
>> TRAIN: [0 1] TEST: [2]
>> TRAIN: [0 1 2] TEST: [3]
>> TRAIN: [0 1 2 3] TEST: [4]
>> TRAIN: [0 1 2 3 4] TEST: [5]
  • 1
    $\begingroup$ Thanks. When we ensemble base models (especially stacking) usually out of fold predictions of different base models are used to train meta model to avoid over fitting. In Time series cross validation, first data point will never be used for testing and last data point will never be used for training. Hence, we loose out of fold predictions for first data point and fitting last point... i.e., In coding example, we have folds [0,1,2,3,4,5] and we get out of fold predictions for [1,2,3,4,5]. Is that fine? $\endgroup$ Nov 19 '18 at 3:15
  • $\begingroup$ @ManimaranSubramanian that is fine. You are right to say that you cannot predict for fold 0. $\endgroup$ Nov 20 '18 at 21:19
  • $\begingroup$ I dont think even time series split cv is appropriate for time series prediction because it trains on those 4 blocks which capture very unique stretches of the variable's movement, in other words 4 market regimes, which are unlikely to re-appear out-of-sample. I think more randomization is needed while still retaining the order of time events. Cant bootstrap resampling with replacement be used within stacking? $\endgroup$
    – develarist
    Apr 14 '20 at 22:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.