I'm currently building a model to predict early mortgage delinquency (60+ days delinquent within 2 years of origination) for loans originating in 2018Q1. I will eventually train out-of-time (on loans originating in 2015Q4), but for now I'm just doing in-time training (training & testing on 2018Q1) -- and even this I've found challenging. The dataset contains ~400k observations, of which ~99% are non-delinquent and ~1% are delinquent. My idea so far has been to use precision, recall, and $F_1$ as performance metrics.
I am working in Python. Things I've tried:
- Models: logistic regression & random forest.
- Model selection: GridSearchCV to tune hyperparameters with $F_1$ scoring (results were not significantly different when optimizing for log-loss, ROC-AUC, Cohen's Kappa).
- Handing imbalanced data: I tried random undersampling with various ratios and settled on a ratio of ~0.2. I also tried messing with the class weights parameter.
Unfortunately, my validation & testing $F_1$ scores are only around 0.1, (precision & recall are usually both close to 0.1). This seems very poor, since with many problems you can achieve $F_1$ scores of 0.9+. At the same time I've heard there's no such thing as a "good $F_1$" range, i.e. it is task-dependent. Indeed, a dummy classifier which predicts proportional to the class frequencies only achieves precision, recall, and $F_1$ of 0.01.
I've tried to find references on what a "good" score for this type of task is, but I can't seem to find much. Others' often report ROC-AUC or Brier Score, but I think these are hard to interpret in terms of business value added. Some report $F_1$ but see overly optimistic results due to data leakage or reporting testing performance on undersampled data. Finally, I've seen some people weight confusion matrix results by expected business costs as opposed to reporting $F_1$, which seems like it may be a better route.
My questions are: (1) is an $F_1$ score of 0.1 always bad?, (2) does it even make sense to optimize for $F_1$ or should I used another metric?, (3) if $F_1$ is appropriate and a score of 0.1 is bad, how might I improve my performance?