1
$\begingroup$

I am recently working on an imbalanced binary classification problem where the data is time ordered. I would like to validate my model using training/validation splits that have the same imbalance ratio (similar to the stratified kfold of Scikit-learn)

I tried the timeseriesSplit of Scikit-learn. However, their implementation does not guarantee that both classes exist with a similar imbalance ratio over the training and data splits. Is there a way to do that?

$\endgroup$

1 Answer 1

0
$\begingroup$

their implementation does not guarantee that both classes exist with a similar imbalance ratio over the training and data splits. Is there a way to do that?

First I would check how the class ratio varies over time. If it does not vary significantly, you don't necessarily need to worry about stratification.

If it does, the way to properly split the data depends on the specifics of what you are trying to do.

$\endgroup$
5
  • $\begingroup$ Thanks for the answer. I checked that but it seems that the scikit-learn Time series split function yields data splits that all are 0 or 1. Therefore I guess a more sophisticated split strategy is needed. $\endgroup$ Mar 9, 2023 at 22:35
  • $\begingroup$ > but it seems that the scikit-learn Time series split function yields data splits that all are 0 or 1 This observation must depend on the number of folds, and it would be very surprising if, whatever value you chose for n_splits resulted in a situation where each of them is all 0s or all 1s. Before you can decide on a split strategy, I would try to understand why the target variable is so seasonal. Once you do, you might be able to find the right features to use and the right split to ensure your cross-validation makes sense. $\endgroup$ Mar 9, 2023 at 23:21
  • $\begingroup$ My problem is a binary classification problem and the target data is a binary variable. The data is time-ordered. $\endgroup$ Mar 10, 2023 at 6:25
  • $\begingroup$ I understand that. What I was asking is if you group your data by time (maybe day or week) what is the mean value of the target variable within each group? Does it vary a lot with time or is it roughly constant? $\endgroup$ Mar 10, 2023 at 15:13
  • $\begingroup$ Be careful: stratification is a variance reduction technique and you might as well suppress "true" variance in your dataset $\endgroup$
    – Ggjj11
    Dec 27, 2023 at 23:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.