1
$\begingroup$

I was thinking. Would it be a good approach to check your features one by one (assuming you have a manageable amount of them) and see the relationship they have with your target variable, if they have a non linear relationship then transform each of those features using their appropriate function for each case to make them linear? In my mind if you do this your are guaranteed to have a better Linear model and also you are able to perform hypothesis testing on each feature to see the relevance of them, giving you the chance to perform some feature selection as well.

I know that the interpretability of model will be thrown out of the window, but the model will give a much better performance. Basically you could potentially end up with a model with only engineer features (assuming that all of them have a non linear relationship)

Would this approach be acceptable and it is worth exploring?

$\endgroup$
  • $\begingroup$ I'm not sure what you mean by transforming non linear data to linear. Modifying therelationship between your features and your target variable might make your model predictions to be point wrong. You might mean, in the other hand, to encode your data in such a manner that it makes it easier for your model to pick up the relationship. For instance, non numerical data that might potentially exhibit a linear relationships to your target. In that case yes, it would help. $\endgroup$ – Baleato Aug 3 '19 at 11:03
0
$\begingroup$

Your idea is good, but you are not the first with this idea.

You can use Generalized Additive Models (GAM) with regression splines to check and/or add non-linearity in a linear regression setup. There is a clear advantage over looking at just descriptive figures one-by-one (with manual feature generation), since you can estimate a whole model with extreme flexibility.

Alternatively, you can simply do a linear regression with a lasso penalty. Add polynomials to your $X$, and let the lasso „shrink“ irrelevant features to zero.

The book „Introduction to Statistical Learning“ covers these topics in Sections 6,7.

BTW: even with polynomials, your model can retain interpretability if you care for this. Polys are relatively easy to interpret. However, under a GAM approach, interpretation is a little more difficult.

Maybe as a note: you want to make features linear (which can be a problem since you can only apply linear transformations to the data). The approaches proposed above aim at making your (linear) model more flexible to cope with non-linearity in data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.