2
$\begingroup$

It is very normal for data scientists and modeling professionals to be concerned with the stability of the model. It basically means that if a variable is important today, it cannot change its importance over time because this instability would be a demonstration of the model's weakness.

It is difficult to argue that stability is good. But in the area of ​​credit modeling (which produces credit scores, for example) people are cracked by stability, to the point of preferring models with less discrimination to a more stable one.

My question is what is the true utility of stability and what is the correct way to make this trade-off with the discriminatory metric of the model (KS, Gini, AUC, etc.).

I would guess (personal guess) that stability somehow ties the model to a less than ideal metric because it does not allow behaviors to change over time. If they don't really change, there will be no problems, but if the effects of the variables are changing over time, it is normal to assume that this is reflected in less stability and analysts work around this by placing simpler models or removing the variables.

$\endgroup$
1
$\begingroup$

I think there might be a bit of confusion here: what is usually called a stable model is a model for which the performance doesn't vary (or not significantly) when sampling a different subset of training data or test data. In other words a model is stable if chance doesn't affect its performance. Typically one can use cross-validation in order to assess the stability of a model: if the variance is high between different splits then the model is unstable. Unstability is often a sign of overfitting, so one should be very cautious with an unstable model even if it seems to perform well: the risk is that the model might obtain good performance on the test set by chance, which means that it will actually perform poorly on new data but there will be no way to detect the problem. This is why people might favor a stable model to a high performance in some applications, because the consequences of deploying an unstable model in production would be costly.

But what you are talking about is not general model stability, it's stability of the performance across time for time-based data. That's a completely different question, it's about whether the model can represent time variations accurately. There are many types of models which can do that, but of course the more complex the model the more data it needs and the higher the risk of overfitting.

$\endgroup$
3
  • $\begingroup$ Thanks for the answer. My point is less about the stability of predictions (something like variance of predictions) and more about the need for stability of predictions over time (described in your second paragraph) and more specifically about the need for the "weight" of variables don't change over time (not just predictions) $\endgroup$
    – sn3fru
    Feb 14 at 12:23
  • $\begingroup$ @sn3fru this completely depends on the kind of model, it's totally possible for a model to allow the "weights of variables" (parameters of the model) to change over time and to have good performance as well. So in general there's no trade-off between stability across time and performance, it's a matter of using the right kind of model. $\endgroup$
    – Erwan
    Feb 14 at 18:33
  • 1
    $\begingroup$ @Erwan: In credit modelling underlying processes are rather slow. So it's really common to use simple model with very constant parameters and choose them to garantee some model stability (stability of outputs, parameters and thus explainability). $\endgroup$
    – lcrmorin
    Feb 15 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.