1
$\begingroup$

Background: I have a dataset that is generated every month (it is similar with card data that contains card demography and transactions every month and new accounts can be added in the middle of data series). From those historical data, I need to build a classification model to predict a binary label for the next month.

Question: Which better cross-validation split type that can be used to get a fair model score assessment (not bias and low variance)? To make it clear, lets take 15 months training data and needs to hypertune the model with 5-folds cross-validation split. I have two options below, but it is ok if you have other.

1. Time series with leave one out type

  • fold 1 : training [1 2 3 4 5 6 7 8 9 10], test [11]
  • fold 2 : training [1 2 3 4 5 6 7 8 9 10 11], test [12]
  • fold 3 : training [1 2 3 4 5 6 7 8 9 10 11 12], test [13]
  • fold 4 : training [1 2 3 4 5 6 7 8 9 10 11 12 13], test [14]
  • fold 5 : training [1 2 3 4 5 6 7 8 9 10 11 12 13 14], test [15]

2. Time Series with leave rest out type

  • fold 1 : training [1 2 3 4 5 6 7 8 9 10], test [11 12 13 14 15]
  • fold 2 : training [1 2 3 4 5 6 7 8 9 10 11], test [12 13 14 15]
  • fold 3 : training [1 2 3 4 5 6 7 8 9 10 11 12], test [13 14 15]
  • fold 4 : training [1 2 3 4 5 6 7 8 9 10 11 12 13], test [14 15]
  • fold 5 : training [1 2 3 4 5 6 7 8 9 10 11 12 13 14], test [15]

Thanks for your answer, will appreciate so much any respond.

$\endgroup$
1
$\begingroup$

Since you want to build a binary classifier based on time-ordered tabular data, I see two possible approaches among others:

  1. as you suggest, split your dataset in ordered train-test folds, so you reproduce the "real" situation of having, at each time interval, a historic dataset to train on and a test (and later evaluation) set; you can use the scikit-learn TimeSeriesSplit to get this type of split, which is similar to what you propose but having always the same test set volume of data:

enter image description here

  1. Reframe your dataset as a usual classification problem, where each sample row has some aggregated information (let's say for a client) like mean, min, max... values of the client attributes, and a binary label; with this frame, apply a k-fold (10-fold is a frequent option) cross validation strategy, you can also check in this answer

By the way, you model should reach a good bias-variance trade-off, rather than a perfect "no bias" model.

$\endgroup$
5
  • $\begingroup$ There is no issue with module to split or feature engeneering. I can do that in a minute. My issue now, is there any statistical reason why you choose cross validation split type 1 rather than type 2? $\endgroup$
    – zonna
    Sep 14 at 20:14
  • $\begingroup$ I would not go for your option 2, just in case the option 1, but the standard method I know and applied sometimes is using a fixed length for the test set, while the train set increases as you have more data. Nevertheless, I would also check if this is what you mean for your use case (i.e. I suggest to check my option 2) $\endgroup$
    – German C M
    Sep 14 at 20:39
  • $\begingroup$ 1. Actually in my current situation I use split type no 1. But there is a feeling my assessment fall high variance score. As my result of cross validation score is higher than my two testing score which is , in this example, month-16 and month-17. That's why I try to figure out different cross-validation split to deal with it. (I'm sure my method is not overfitting). 2. Your options 2 is already done in feature engineering part. Now we are in modeling part. $\endgroup$
    – zonna
    Sep 15 at 2:25
  • $\begingroup$ then, if you reframe it with my second option, you do not need to worry about time series split format; if I understant, you would already have a client info per row format; maybe you can give more info on this $\endgroup$
    – German C M
    Sep 15 at 7:08
  • $\begingroup$ Yes, you are right. I already have those information in one row. What I wonder here is for example in the month-11th, I have 1000 client with 990:10 for label 0:1. And in the month-12th, I have 1200 client with 1100:100, it keep increasing for the next month (sometimes decreasing). This variance of total client number and weight for label 0 and 1 make me not confidence with the result of cross-validation with splitting type 1. I am afraid the result is bias and have high variance. $\endgroup$
    – zonna
    Sep 16 at 9:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.