I know that having correlated attributes violates the linear model assumption of independent attributes, and I'm not interested in creating a more sophisticated model to tease apart the dependent effects of various attributes on each other. Rather, I'm trying to figure out if there is a way to make the assignment of coefficients more consistent when dealing with correlated attributes in a linear model. This simple example should help to explain:
Lets say you have two perfectly correlated attributes (r = 1), which also have a meaningful correlation to the target variable. In this case, you could set the coefficient of either attribute to zero and still explain all of the variation with the other attribute. It's arbitrary which coefficient is set to zero, because the error is still being minimized by the same amount either way. And in fact there are an infinite number of solutions because there are an infinite number of combinations of coefficients that still produce the correct sum. This makes sense, because the two attributes are essentially one attribute: they have the same values at each point. So mathematically there is no reason to select one solution over any other. However, if you're performing an analysis to determine the relative contribution of each attribute to the target variable, you may want to start with a null hypothesis that each attribute contributes an equal amount, rather than choosing an arbitrary level of contribution from each attribute.
Of course real world data would be more messy than this toy example and have weaker correlations, but the fundamental problem still exists: correlated attributes have arbitrary levels of contribution in proportion to their correlation. If the data represents something substantial in the real world and could plausibly have a causal effect on the target variable, it doesn't make sense to arbitrarily set it's influence (coefficient) to a low value just because it's correlated with another attribute that arbitrarily contributes more of that correlated variance towards the target variable. We simply don't know the causal relationships between these variables from this data, so we can't prioritize one variable over another, but we can at least try to assign equal contribution as a consistent default solution.
One potential approach is to perform some kind of dimensionality reduction and combine our attributes together, which of course adds explanatory power. But even here, I'm not so sure that something like PCA would be less arbitrary in the relative contribution of correlated attributes to a component. You could still set a low weight to one of those attributes. A table of correlation coefficients would also provide some much needed explanatory power, but not resolve the issue of learning inconsistent weights in the linear model based on minor changes in the training data.
In one training set, one of the two correlated attributes may explain slightly more variance in the target variable and be given a much larger weight, whereas in another training set, the opposite could occur. In some ways this is a kind of over-fitting, so regularization comes to mind as a possible solution. While I do think that regularization could help to prevent weights from being set to zero, I don't know of a regularization technique that addresses this issue directly, so I came up with an idea. Let's say you want to enforce this assumption of a null hypothesis where correlated attributes both contribute equality in proportion to their degree of correlation. For example, perfectly correlated attributes would have to have equal coefficients, and attributes with no correlation to any other attributes could be set to any value without any constraint. I'm not sure exactly how to formulate the constraint for partially correlated attributes, and I'd be curious to hear any suggestions.
Enforcing this constraint should make it so that the coefficients aren't as sensitive to minor changes in the training data, so I'd like to hear what people think about it as a regularization technique that addresses this specific issue. Does something like this already exists? Is it worth looking into as a useful technique? More consistent weights means more consistent interpretation of those weights, even if it relies on an additional known caveat. Getting back to the use case of analyzing the relative contribution of different attributes to the target variable, one might say, "these two attributes were highly correlated, so our model assigned some contribution of each to the target variable. We can think of these coefficients as representing something like 'potential impact' that can be assessed further for causal connection." What do you think? Is this a helpful approach for dealing with an actual fundamental source of inconsistency with linear models, or does it do more harm than good?