2
$\begingroup$

I have a dataset made of roughly 100 time-series and my final goal is to obtain a classification of each point (detection problem). To do so I have labels so I decided to use an XGB model to perform the detection over some features that I have created. The time-series are not sampled uniformly and the time-order looks not so important for this specific problem so far.

The problem is that when I perform the StratifiedKFold (as per Sklearn) the results looks promising and the standard deviation of the relevant metrics among the kfold is really small. Nevertheless, if I remove one time-series entirely from the training set and fitting the model over the other ones I am not able to replicate the same results.

This gap between Kfold performances and "real test" looks to me like the training is not really generalising the problem, despite the good results during the Kfold validation.

Do you have any idea to fix this problem? or any advice?

Improve this question
$\endgroup$
2
  • $\begingroup$ Try using another model which generalizes better? 100 samples sounds like to little for XGB $\endgroup$
    – Carl Rynegardh
    Mar 6, 2019 at 18:53
  • $\begingroup$ Yes, it is always possible to reduce the variance of the model but just two points: 1) I did not say that I have 100 points I said 100 different time-series of the same problem with many points inside. 2) why the model perform good on Kfold validation, since it is a kind of generalization test. $\endgroup$
    – FrankBool
    Mar 6, 2019 at 19:11

1 Answer 1

Reset to default
3
$\begingroup$

Using normal KFold cross-validation for time-series data will yield a highly optimistic error estimate since you are using data from the future to predict the past. The model just has to learn to interpolate, not to predict. Therefore you have to use time-wise CV:enter image description here

Furthermore, if your goal is to predict the future for a time-series you do not know the past of, you have to use leave-one-group-out, time-wise-CV to get a realistic performance estimate: Train on all time-series except for a test-set of time-series, but only using data up to a certain point in time. Validate on the test-time-series but only using data after the timepoint.

I doubt you will have success with this approach since this is often a very difficult problem.

For choosing the right cross-validation method you have to be clear about the goal of your method.

  1. Do you want to interpolate missing data for a couple of time-series? Use K-Fold CV

  2. Do you want to predict the future of some time-series, for which you know the past? Use time-wise CV

  3. Do you want to predict the future of some time-series, but you only know the past of some other time series? Use Leave-one-Group-out, time-wise CV

I assume you are after the second point since this is the most common problem.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.