I'm interested in modeling the first variable using the rest as explanatory. A simple OLS yields a more or less satisfying model but inuitively, i know that there are k (between 2 and 4) regimes where the correlation between my Y and certain variables switch completely making the one full OLS across the dataset sort of irrelevant as we sort of smooth all of these regimes by doing this. I attempted differentiating the regimes using correlation similarity with a k-means algorithm in R, the result is the coloring in the pictures. i was hoping that dividing the dataset this way would reveal more "straight lines" where the OLS would be more "fitting". I wonder if i'm going in the right way, would be grateful if anyone could point to better ways at achieving the fundamental problem : Modeling Y~X while taking into account that there are different regimes which require a likely different coefficient or (model altogether ?) EDIT : I also tried MARS models but the result is not satisfying at all from a "physics" sense of how these variables are supposed to actually work together + changing the training set slightly yields changes the coefficients too much.
Just a couple thoughts:
- It looks like these "regimes" could be represented as a latent variable: you could probably design a bayesian model in which the OLS model depends on the value of this latent variable. This means that the model would still be trained only with the observed features, but would internally predict the value of the regime and this value would determine the parameters of the OLS model.
- A more direct approach for this kind of case-by-case setting would be to use decision trees (or random forests), since they can handle independent models in different branches. However I'm not sure how to make decision trees and linear regression work together (or if it's possible at all).