i don't have a formal background in this so please bear with me. This is what my dataset looks like : clas

I'm interested in modeling the first variable using the rest as explanatory. A simple OLS yields a more or less satisfying model but inuitively, i know that there are k (between 2 and 4) regimes where the correlation between my Y and certain variables switch completely making the one full OLS across the dataset sort of irrelevant as we sort of smooth all of these regimes by doing this. I attempted differentiating the regimes using correlation similarity with a k-means algorithm in R, the result is the coloring in the pictures. i was hoping that dividing the dataset this way would reveal more "straight lines" where the OLS would be more "fitting". I wonder if i'm going in the right way, would be grateful if anyone could point to better ways at achieving the fundamental problem : Modeling Y~X while taking into account that there are different regimes which require a likely different coefficient or (model altogether ?) EDIT : I also tried MARS models but the result is not satisfying at all from a "physics" sense of how these variables are supposed to actually work together + changing the training set slightly yields changes the coefficients too much.


Just a couple thoughts:

  • It looks like these "regimes" could be represented as a latent variable: you could probably design a bayesian model in which the OLS model depends on the value of this latent variable. This means that the model would still be trained only with the observed features, but would internally predict the value of the regime and this value would determine the parameters of the OLS model.
  • A more direct approach for this kind of case-by-case setting would be to use decision trees (or random forests), since they can handle independent models in different branches. However I'm not sure how to make decision trees and linear regression work together (or if it's possible at all).
|improve this answer|||||
  • $\begingroup$ I also thought of this after posting the issue. I was thinking of designing (somehow) a function which predicts whether we are in regime 1 or 2 and then use it as an indicator function with 2 OLS functions trained on their own regime something like : OLS_1(X)*1{prob we are in regime 1 > prob we are in reg 2} + OLS_2(X)*1{...} . does this sound good and is there some pre made model that does this painlessly ? thanks $\endgroup$ – Saad Filali Khattabi Jan 2 at 22:29
  • $\begingroup$ @SaadFilaliKhattabi I was going to tell you that unfortunately I don't know any model specifically like that, but I googled around out of curiosity and I found this which might be close to what you need: statsmodels.org/dev/examples/notebooks/generated/… $\endgroup$ – Erwan Jan 2 at 22:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.