on the back of this topic (When to remove correlated variables) I feel a follow up is needed, with the focus here being on raw performance and risk of distribution shift.
assuming little to medium interpretability is needed, we have a scenario when even medium-ish gains in predictive performance are very very welcome but, in order to do so, we have to step up the number of moderate-to-strongly correlated variables (
0.6-0.8 range, linear correlation). this provides benefits (using
R2 here) in the order of 5 to 10 percentage points. these gains absolutely do make a difference for the problem at hand.
but I am concerned that I am implicitly tethering to the same phenomenon (therefore making the model less robust) especially when distribution shifts are on the horizon for the variables upon which the model relies the most.
so, do you have any approaches or even rule of thumbs that can be generally applicable in these scenarios?