I know that popular linear regression models such as Lasso or Logistic Regression are widely used in practice because they perform reasonably good, are efficient and interpretable.

As far as I know, the only way for these models to learn non-linear relationships between $X$ and $y$ is by applying non-linear transformations to $X$ through basis functions $\phi(X)$ (as it is summarized in the scikit-learn documentation).

I wonder if this approach is actually used in real world problems, where usually we don't have any prior information about how to design $\phi$.


Yep, that's a thing. It's called a "Generalized additive model (GAM)":


You may also be interested in "multivariate adaptive regression splines (MARS)":


EDIT: Regarding demonstrating that these are actually used, I'm not sure what you're looking for. I've never seen a survey to try and gauge the popularity of specific models, and I'm not sure how useful such a thing would be. I could speak from my subjective experience, but I don't consider that particularly meaningful either.

If you just want examples:

For context:

Take these counts with a solid dose of salt: in my experience random forests are in much wider use in practice than neural networks. But like I said earlier, you shouldn't really trust my subjective experience either since that's just anecdotal.

| improve this answer | |
  • $\begingroup$ But to what extent do you think they are actually used in the industry? I would really appreciate some examples. $\endgroup$ – hipoglucido Jan 23 '18 at 11:26
  • $\begingroup$ It's worth knowing about, as an extension of linear/logistic regression. $\endgroup$ – Emre Jan 23 '18 at 21:11
  • $\begingroup$ Linked to examples on PubMed, contextualized count of hits for models by comparing with hits for RF and DNN. $\endgroup$ – David Marx Jan 24 '18 at 5:18

In responses to the comment requesting real world application. I have not found models like GAM used particularly often in the industry, but there were notable exceptions in my career.

Insurance companies, especially life insurance companies, tend to prefer an explainable model over one that just makes good predictions. In this space, I have used GAM to model spatial distribution. I can't go into particular details, but to give you a related example, imagine an experiment design regarding the survey of fish. A properly designed survey will stratified an environment; however, within the environment there exist covariates. The traditional approach would be at apply a GLM and add covariates into the model. However, it's easy to see that some predictors may not be linear. For example, fish are drawn to different temperature, so the relation for the environment of the water and the temperature of the wish should probably not be modeled in a linear fashion. Thankfully we could use a cubic spline instead and produce a readily interpretable result.

Another example, would be in credit scoring. In practice, there's usually a myriad of mixed models working together to produce a credit score and there exist different types of models for different level of products and risk. Generically speaking though, some of these models can be hard to understand and find what driving factors effect it. MARS can be used as a good baseline piecewise model for credit scores.

In short, it's like any other model. There's a use, time and place for it. I've found that companies that rely heavily on more traditional statistics and process understanding are more inclined to use models like GLM or GAM, while newer tech heavy companies favor the more blackbox quick iteration push to production approach. Neither is better than the other, just that each industry has its own standards and goals.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.