Imbalanced Cost-Sensitive Learning Workflow - How to split the data, tune hyperparameters and apply adecision threshold?

Question

I am facing a problem with imbalanced dataset in which I would like to detect the rare event. My questions are more of general strategy about the whole workflow and I would like to hear your thoughts suggestions.

Data budget:

How should i split my dataset into training/validation(?)/threshold tuning(?)/test set? I will expand on this question as we go.

Hyperparameter tuning using cross-validation:

I think based on this post that it would be more wise to set the objective of the hyperparameter tuning algorithm (e.g. Optuna) to be a loss function (e.g. brier score loss) rather than a hard class metric like f1-score. The reason is that a hard class metric like f1-score would be optimised based on a decision threshold of 0.5 which would probably require tuning later.

Another matter which should be considered during this phase are the class weights which are more preferable to upsampling/downsampling techniques based on my understanding of these 1, 2. Would it be wise to balance them out and be done with it or tune them as well using example pos_weight for XGBoost?

Decision Threshold:

After finishing with with the hyperparameter tuning, the next phase would be to tune the decision threshold (not actually necessary but requested by the decision-makers usually) for example by using the sklearn's TunedThresholdClassifierCV and the desired metric (e.g. f1-score). In order to do that we would need an extra set of data other than the training data which we can call ThresholdTuning set. How large should that be based on your experience?

Would it make sense to use the same set to perform feature selection using sklearn's permutation_importance? Should the scoring of the permutation importance be based on the brier score loss once again?

Model validation:

After hyperparameter tuning and threshold tuning, would it make sense to proceed to testing using the test set or should a validation set be included as well prior to that? What % of the initial dataset should the validation and test set be in this case especially in cases of small amount of data (say 5000)?

Sorry for the long post, I look forward to hearing your thoughts/suggestions.

Answer 1

1. You should definitely tune with Brier Score or log-loss. Tuning on an accuracy-like metric with rare events is often useless because very few data points will have P(Y=1|X) above .5 (unless the event is extremely predictable).

2. In my experience with rare event modeling, I have not found class weights to be useful. Your experience may vary but I would focus first on getting the main train-and-evaluate loop written before messing around with class weights. Once you have your main loop established, you can test whether adjusting the class weights makes a difference.

3. Assuming you have a train/test split, then you don't need another dataset to tune the decision threshold because sklearn uses a cross-validated approach to select the optimal threshold. You would the train the model on the training data and then apply TuningThresholdClassifierCV to the testing data. The loss-value that you get from this process will be a slightly over-optimistic (eg L(M,k*) on the testing data will be slightly lower than L(M,k*) applied to brand new data, where M is the model and k* is the threshold chosen by Tuning on the test set ) but the degree of over-optimism should be very low.

4. Based on your post, it seems like your strategy is a train/test split and then you will select a model using cross-validation on the training set. That's a totally fine set-up so long as you're very careful to only fit on the training data (and avoid things like leakage). Some people advocate having a train/test/validation split to avoid the potential of overfitting to the testing data (eg you see that your prediction error on the test set is high, so you try a different model, etc) but the practical degree of over-fitting that occurs from that has been shown to be quite low (like in this paper: https://proceedings.neurips.cc/paper/2019/file/ee39e503b6bedf0c98c388b7e8589aca-Paper.pdf). I would do a standard 80/20 split.