I have a dataset made of roughly 100 time-series and my final goal is to obtain a classification of each point (detection problem). To do so I have labels so I decided to use an XGB model to perform the detection over some features that I have created. The time-series are not sampled uniformly and the time-order looks not so important for this specific problem so far.

The problem is that when I perform the StratifiedKFold (as per Sklearn) the results looks promising and the standard deviation of the relevant metrics among the kfold is really small. Nevertheless, if I remove one time-series entirely from the training set and fitting the model over the other ones I am not able to replicate the same results.

This gap between Kfold performances and "real test" looks to me like the training is not really generalising the problem, despite the good results during the Kfold validation.

Do you have any idea to fix this problem? or any advice?

  • $\begingroup$ Try using another model which generalizes better? 100 samples sounds like to little for XGB $\endgroup$ – Carl Rynegardh Mar 6 at 18:53
  • $\begingroup$ Yes, it is always possible to reduce the variance of the model but just two points: 1) I did not say that I have 100 points I said 100 different time-series of the same problem with many points inside. 2) why the model perform good on Kfold validation, since it is a kind of generalization test. $\endgroup$ – Frankshore Mar 6 at 19:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.