Is an $F_1$ score of 0.1 always bad?

Question

I'm currently building a model to predict early mortgage delinquency (60+ days delinquent within 2 years of origination) for loans originating in 2018Q1. I will eventually train out-of-time (on loans originating in 2015Q4), but for now I'm just doing in-time training (training & testing on 2018Q1) -- and even this I've found challenging. The dataset contains ~400k observations, of which ~99% are non-delinquent and ~1% are delinquent. My idea so far has been to use precision, recall, and $F_1$ as performance metrics.

I am working in Python. Things I've tried:

• Models: logistic regression & random forest.
• Model selection: GridSearchCV to tune hyperparameters with $F_1$ scoring (results were not significantly different when optimizing for log-loss, ROC-AUC, Cohen's Kappa).
• Handing imbalanced data: I tried random undersampling with various ratios and settled on a ratio of ~0.2. I also tried messing with the class weights parameter.

Unfortunately, my validation & testing $F_1$ scores are only around 0.1, (precision & recall are usually both close to 0.1). This seems very poor, since with many problems you can achieve $F_1$ scores of 0.9+. At the same time I've heard there's no such thing as a "good $F_1$" range, i.e. it is task-dependent. Indeed, a dummy classifier which predicts proportional to the class frequencies only achieves precision, recall, and $F_1$ of 0.01.

I've tried to find references on what a "good" score for this type of task is, but I can't seem to find much. Others' often report ROC-AUC or Brier Score, but I think these are hard to interpret in terms of business value added. Some report $F_1$ but see overly optimistic results due to data leakage or reporting testing performance on undersampled data. Finally, I've seen some people weight confusion matrix results by expected business costs as opposed to reporting $F_1$, which seems like it may be a better route.

My questions are: (1) is an $F_1$ score of 0.1 always bad?, (2) does it even make sense to optimize for $F_1$ or should I used another metric?, (3) if $F_1$ is appropriate and a score of 0.1 is bad, how might I improve my performance?

Answer 1

From a credit scoring point of view : a $F_1$ score of $0.1$ seems pretty bad but not impossible with an unbalanced data-set. It might be enough for your needs (once you weight your errors by the cost). And it might not be possible to go higher (not enough data to predict an event that appears random). In credit scoring there is always a 'random' part in the target (sudden death, divorce ...) depending on the population and the goal of the loans.

1. You might want to investigate your features and your target. Basically : statistically, on an univariate approach, do you have features that appears predictive of the target ? (Age of the person ? revenue ? purpose of the loan ?). You might also need to investigate the target : do you have some questionnaire that would allow to get an insight on why the person defaulted ? (If the majority of default come from random event, you might not be able to modelise it).

2. The main problem with $F_1$ score in credit scoring is not data imbalance, but cost imbalance. Type I and Type II errors have far differents consequences. Given that you already gave the loans I am not even sure there is a cost associated with false positive (saying someone will default when it won't). It might be interesting to weight precision and recall (i.e. use $F_\beta$ as defined here). Another problem is that it is usually good for a binary decision. Depending on what you want to use the model for (measuring risk of already granted loans ? granting new loans ? pricing new loans ?) there might be alternatives that better capture model discrimination (AUC - see its statistical interpretation) or individual % chance of default (Brier Score).

3. Assuming that there is no specific problem with your current modelling (Feature engineering, imbalance treatment, 'power' of your model). There are some credit-scoring specific things you can do. Work on your target definition (what if you do 90+ days delinquant in the 5 years after origination ?). Try to collect more data about your clients and their behavior (purpose of the loan, others products they use at your bank... etc.).

Answer 2

(1) For the sake of keeping it short in your case: yes 0.1 is bad. To avoid philosophical discussions let's just assume you have to get this higher.

(2) It definitely makes sense since your dataset is highly imbalanced. Do not expect to have one metric where you fail miserably and on the other one, you succeed. That's not how it works, they are most often correlated.

(3) This is a very General Question. Do more machine learning. But here are some propositions worth exploring: up-sampling, more complex models (lgbm, nn), feature Engineering (understand your data!), Analyse the failed predictions here you can see what you need to improve etc...

Finally, I just want to make it clear that not everything can be modeled. Maybe from your dataset you cant make this conclusion. This is for you to decide when you exhausted all of the possible potential solutions.

Answer 3

The short answer is yes. Nonetheless you should have been deeper while data understanding process i.e analyzing if there are really features that separate/differentiate the good payers vs the delinquent ones.

Say for example you have numeric variables such as current balance, number of delinquent accounts, number of inquiries in the last six months,etc If you plot the distribution of those features based on the class, are those distribution different (you could use KS score to validate that)?

The point is,if there is no "clear" difference between the characteristics of both populations (payers vs non payers) no matter if you have 50 - 50 target distribution, it will be hard for any model to separate the classes, but of course the fact of imbalance makes things harder.

Another point to be consider is that if it is possible for you to redefine the metric, so that there might be an earlier indicator of delinquency, it might help to balance your sample.

I mean suppose you label those who did not pay its obligations after one year as delinquent, how related is this event with the one on which a user has not payed from the first 6 or 7 months? In this sense you could redefine your metric with one that is highly anticipating the fact that the user is going to be delinquent and this might change the distribution of your classes so that there will be more cases from the non payers (assuming this is the minority class)

One last thing I want to mention is you could give a try on one class classification as studied here

Good discussion on the topic is here

Answer 4

I have a domain observation, rather than a modeling one. It's based on my 2007 analysis of 125K securitized subprime loans originated in 2006 by a single issuer with a broker, yield spread marketing model. In addition to 50 origination variables, monthly patterns of payments were tracked.

1. Delinquency is not solely a matter of credit underwriting, but of the entire lending decision. Other components include

a. net and coupon coupon (index changes in ARMs affects ability to pay); b. assumptions about the stability of the housing market; c. balance between the acceleration of recognition of gain on sale and the time-discounted loss on retained risk; d. the influence of moral hazard on rigor of credit underwriting (e.g., degree of skepticism on borrower representation of intent to occupy as a permanent residence); e. life events, such as loss of employment, illness of an income credit, loss of employment (involuntary to to care for a family member, for example), business failure for self-employed or divorce; and f. exogenous events, such as the sudden increase in an ARM index that increases debt service burden.

3. The assessment of all these factors at origination was made in part on the basis of automated underwriting systems. The AUS assessments depended on the algorithms, that were developed based on historical data. In my case, the historical data reflected performance of a much smaller pool in a much more stable market. It also embedded a strong reliance on "FICO" credit scores as a predictor of loan performance. In isolation FICO had no correlation with loan default for the 2006 pool, in the event.

4. Another portion of the assessment was based on prescriptive manual underwriting guidelines. Although the guidelines were "objective", they were also complex, ambiguous and difficult to apply. The sheer volume of applications to be processed degraded the quality of the application of the guidelines and impaired the application of judgments required in borderline cases.

5. Following initial underwriting both favorable and unfavorable decisions were subject to a second round of review by management. Moral hazard, induced by volume-based incentive compensation, resulted in more negative than positive recommendations being reviewed and the expected result is that any rebalancing of factors judgmentally represented an increase, rather than decrease of risk.

6. All of the loans in the pool made the first scheduled payment. A small, but still unusually large number failed to make a second or third payment and were repurchased as required by contract. In survival analysis terms, these loans were "censored." It was generally understood that such loans represented failure of underwriting.

7. Beginning with the fourth scheduled payment, any assessment of the underwriting process was doubly affected by surviver bias. None of the loan applications that were unsuccessful under the same underwriting guidelines could be assessed for performance. Unless it is assumed that all of such loans would necessarily have become delinquent, inferences drawn from performance of loans under the process are weakened.

8. During the course of a loan's life it may become periodically delinquent without defaulting. For example, a loan may miss two consecutive payments, catch up, and then miss another two.

9. A loan that misses three consecutive payments is in default, goes into the foreclosure/liquidation process. However, prior to resolution, the loan may reinstate or a forbearance/repayment plan be arranged, in which case the loan agains become subject to the possibility of delinquency.

10. The more often a loan become delinquent without being fully liquidated, the weaker the association between the underwriting decision and the occurrence of any default.

11. Another complication that is not present in the 2006 pool occurred in other pools of the same vintage in which one originator sells a closed loan to a different lender. During the period in which the transfer of servicing takes place, payments may be delayed in being recorded and loans will be classified as delinquent incorrectly.

12. In terms of the association between the occurrence of any delinquency and the origination processes, it seems likely that connection is highly variable.

13. All of these considerations suggest that the influence of origination variables on first delinquency is a) affected by variation within the origination variables in terms of application, b) fluid weights attached to the origination variables and c) the duration between origination and the date of first delinquency and d) the ultimate economic consequences of any single delinquency differ.

14. The pool you are assessing, Fannie originated loans from 2018 are different in both the underwriting criteria and application. Based on my experience with Fannie's program before 2005, loan approval was much more automated, much less subject to "manual override", and policed by Fannie's market power to force repurchase of delinquent loans from originators and to punish bad actors with punitive "G-fees" or the "death penalty" of expulsion from program approval.

15. Still, the Fannie AUS was a black box with an algorithm that could only be estimated to understand the degree of play available to qualify a loan through fine tuning of qualification criteria within the rules--gaming the system.

16. Therefore, while Fannie loans were subject to a narrower range of uncertainty as to the measurement and weighting of the independent variables, considerable uncertainty in their predictive power for delinquency, let alone the timing of first delinquency remains.

17. My observations are in part based on the analysis I performed, and on judgment based on over 15 year's experience as a mortgage-backed securitization lawyer. I was responsible for over $150 billion of mortgage loan and ended my career as Senior Vice President/Associate General Counsel of Washington Mutual Bank. That experience does not enhance the quality of the data analysis that I performed. It may suggest that the processes determining origination are highly stochastic and should be taken into account in any model to predict a binary outcome of a delinquency during any given period.