I have a dataset made of roughly 100 time-series and my final goal is to obtain a classification of each point (detection problem). To do so I have labels so I decided to use an XGB model to perform the detection over some features that I have created. The time-series are not sampled uniformly and the time-order looks not so important for this specific problem so far.
The problem is that when I perform the StratifiedKFold (as per Sklearn) the results looks promising and the standard deviation of the relevant metrics among the kfold is really small. Nevertheless, if I remove one time-series entirely from the training set and fitting the model over the other ones I am not able to replicate the same results.
This gap between Kfold performances and "real test" looks to me like the training is not really generalising the problem, despite the good results during the Kfold validation.
Do you have any idea to fix this problem? or any advice?