Linear Model With Highly Correlated Attributes Producing Inconsistent Weights

Question

I know that having correlated attributes violates the linear model assumption of independent attributes, and I'm not interested in creating a more sophisticated model to tease apart the dependent effects of various attributes on each other. Rather, I'm trying to figure out if there is a way to make the assignment of coefficients more consistent when dealing with correlated attributes in a linear model. This simple example should help to explain:

Lets say you have two perfectly correlated attributes (r = 1), which also have a meaningful correlation to the target variable. In this case, you could set the coefficient of either attribute to zero and still explain all of the variation with the other attribute. It's arbitrary which coefficient is set to zero, because the error is still being minimized by the same amount either way. And in fact there are an infinite number of solutions because there are an infinite number of combinations of coefficients that still produce the correct sum. This makes sense, because the two attributes are essentially one attribute: they have the same values at each point. So mathematically there is no reason to select one solution over any other. However, if you're performing an analysis to determine the relative contribution of each attribute to the target variable, you may want to start with a null hypothesis that each attribute contributes an equal amount, rather than choosing an arbitrary level of contribution from each attribute.

Of course real world data would be more messy than this toy example and have weaker correlations, but the fundamental problem still exists: correlated attributes have arbitrary levels of contribution in proportion to their correlation. If the data represents something substantial in the real world and could plausibly have a causal effect on the target variable, it doesn't make sense to arbitrarily set it's influence (coefficient) to a low value just because it's correlated with another attribute that arbitrarily contributes more of that correlated variance towards the target variable. We simply don't know the causal relationships between these variables from this data, so we can't prioritize one variable over another, but we can at least try to assign equal contribution as a consistent default solution.

One potential approach is to perform some kind of dimensionality reduction and combine our attributes together, which of course adds explanatory power. But even here, I'm not so sure that something like PCA would be less arbitrary in the relative contribution of correlated attributes to a component. You could still set a low weight to one of those attributes. A table of correlation coefficients would also provide some much needed explanatory power, but not resolve the issue of learning inconsistent weights in the linear model based on minor changes in the training data.

In one training set, one of the two correlated attributes may explain slightly more variance in the target variable and be given a much larger weight, whereas in another training set, the opposite could occur. In some ways this is a kind of over-fitting, so regularization comes to mind as a possible solution. While I do think that regularization could help to prevent weights from being set to zero, I don't know of a regularization technique that addresses this issue directly, so I came up with an idea. Let's say you want to enforce this assumption of a null hypothesis where correlated attributes both contribute equality in proportion to their degree of correlation. For example, perfectly correlated attributes would have to have equal coefficients, and attributes with no correlation to any other attributes could be set to any value without any constraint. I'm not sure exactly how to formulate the constraint for partially correlated attributes, and I'd be curious to hear any suggestions.

Enforcing this constraint should make it so that the coefficients aren't as sensitive to minor changes in the training data, so I'd like to hear what people think about it as a regularization technique that addresses this specific issue. Does something like this already exists? Is it worth looking into as a useful technique? More consistent weights means more consistent interpretation of those weights, even if it relies on an additional known caveat. Getting back to the use case of analyzing the relative contribution of different attributes to the target variable, one might say, "these two attributes were highly correlated, so our model assigned some contribution of each to the target variable. We can think of these coefficients as representing something like 'potential impact' that can be assessed further for causal connection." What do you think? Is this a helpful approach for dealing with an actual fundamental source of inconsistency with linear models, or does it do more harm than good?

Answer 1

I would assume the classical L2 regularization would already do so. In the following, I will give some mathematical reasoning for that.

Perfectly correlated columns

Let's for the sake of simplicity for the moment stick to your extreme case of $r=1$, i.e. the two columns (let's call them $x_1$ and $x_2$) are directly bound by a linear mapping, e.g. $$x_2 = ax_1 + b$$.

If now the weights of $n$ variables are given by $(w_i)_{i=1,\ldots,n}$ and the intercept is given by $w_0$, then the L2-Loss with factor $\lambda$ is given by $$L_{2}=\lambda\sum_{i=0}^nw_i^2$$

Given the linear binding between $x_1$ and $x_2$, we can compensate every change of the weight $w_1$ be an change of the weight $w_2$ and the intercept $w_1$ so that the prediction of the model does not change. Let $w_i^\prime$ ($i=0,1,2$) be the updated weight. If the prediction shall not change, we get: $$\begin{align} w_0^\prime + w_1^\prime x_1 + w_2^\prime x_2 + \sum_{i=3}^nw_ix_i &= w_0 + w_1 x_1 + w_2 x_2 + \sum_{i=3}^nw_ix_i\\ w_0^\prime + w_1^\prime x_1 + w_2^\prime x_2 &= w_0 + w_1 x_1 + w_2 x_2\\ w_0^\prime + w_1^\prime x_1 + w_2^\prime (ax_1+b) &= w_0 + w_1 x_1 + w_2 (ax_1+b)\\ w_0^\prime + w_2^\prime b + (w_1^\prime + aw_2^\prime)x_1 &= w_0 + w_2 b + (w_1 + aw_2)x_1 \end{align}$$

From this we get: $$ w_0^\prime + w_2^\prime b = w_0 + w_2 b \quad\wedge\quad w_1^\prime + aw_2^\prime = w_1 + aw_2$$ or: $$\begin{align} w_2^\prime &= \alpha-\frac{w_1^\prime}{a}\\ w_0^\prime &= \beta + \frac{b}{a}w_1^\prime \end{align}$$

with $\alpha = \frac{w_1 + aw_2}{a}$ and $\beta=w_0 + b(w_2-\alpha)$

This gives us the updated loss as a function of $z=w_1^\prime$: $$L_2(z) = \left(\beta + \frac{b}{a}z\right)^2 + z^2 + \left(\alpha-\frac{z}{a}\right)^2 + \sum_{i=3}^nw_i^2$$

This function has its minimum at $$ z = \frac{a(\alpha - \beta b)}{b^2 + a^2 + 1}$$

From this, one can compute the $w_i^\prime$ that minimize the loss without changing the predictions for a given $w_i$ ($i=0,1,2$).

I will skip the computation of the optimum, here! (since it is just a specific detail and not necessary for the answer).

For the special case $a=1$ and $b=0$, we get the minimal loss for: $$w_0^\prime = w_0, \qquad w_1^\prime=w_2^\prime = \frac{w_1+w_2}{2}$$

Summary

Using the L2-Loss for regularization leads already to a well-defined share between the weights. In case if identical features, this is the average of their features.

Real World

In the real world, we do not have perfect correlation. Still, be minimizing the squared sum of the weights, the L2-regularization pulls the weights of correlated features towards a defined state. Here, the regularization factor $\lambda$ controls how much small predictive gains will pull the weights away from this state (as for every regularization)