I've been working on creating a regularization term that ensures that correlated attributes are given similar weights in a linear model. This helps to avoid some of the inconsistency in the weights of the correlated attributes. For more insight on the problem I'm trying to solve see: this post

Here is the regularization term: $\lambda \cdot r^2 \cdot (B_1 - B_2)^2$

Where $\lambda$ is the regularization parameter, $r$ is the correlation coefficient between the two attributes, and $B_1$ and $B_2$ are the model coefficients of the two attributes. You would need to extend the term for each pair of attributes that you are concerned about, which could be all pairs if you're dealing with a small number of attributes, or just the highly correlated ones.

If the attributes are highly correlated, and their coefficients in the linear model are very different, then a high loss penalty will be applied. Please let me know what you think of this novel regularization term, or if you've seen something like it already.

Edit: Thank you for your feedback on the potential usefulness and flaws of this term. Based on Broele's suggestion, I first need to figure out how to handle negative correlations. Then I can take Luca Anzalone's suggestion of empirically testing the term on some actual data. So besides my broader question of "Is this a useful term for improving the stability and interpretability of a linear model's weights?" I now have the specific question, "How do I modify this term to handle negative correlations?". My first thought is to handle it with cases. If $r$ is negative, then use addition instead of subtraction:

$\lambda \cdot r^2 \cdot (B_1 + B_2)^2$

This means that when there is a high negative correlation, the coefficients should become the negation of each other. However, I'm concerned that this equation might drive all of the weights towards zero. Is there a better way to formulate this regularization term and still achieve the goal of making weights more similar when correlation is high, and similar value but negated with high negative correlation?

  • $\begingroup$ What exactly is your question? We are a question-and-answer site, so we require you to articulate a speciifc question. "Please let me know what you think of this" is likely a bit too vague and open-ended -- see our help center. $\endgroup$
    – D.W.
    Jul 29, 2023 at 8:35
  • 1
    $\begingroup$ You will face trouble with negatively correlated features. $\endgroup$
    – Broele
    Jul 29, 2023 at 10:26
  • $\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$
    – Community Bot
    Jul 29, 2023 at 13:39
  • $\begingroup$ I think you should compare (and maybe also provide the results) the linear model with and without your regularizer $\endgroup$ Jul 29, 2023 at 16:15
  • 2
    $\begingroup$ Do you really want to have both weights equal to each other? Imagine two features with $x_2 = 100 * x_1$? In my answer to you other post (it is more a post then a question), I showed that L2 regularization will already lead to a defined state, which would be influenced by this factor of 100. $\endgroup$
    – Broele
    Jul 29, 2023 at 22:14


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.