1
$\begingroup$

I have a dataset with parameters (features) a,b, c, etc. We need to develop a model to predict a (our target).

b is correlated with a significantly (85%) and I suspect linear dependence. c is a measurement of b in another depth, so it has a high correlation with b and a good correlation with a. Also, there are a bunch of other parameters (numerical features) whose dependence of a is non-linear (from background knowledge and box plot analysis). These features don't have a high correlation with a but have an effect on a.

I am building a model to predict a based on the knowledge above. I think that a multi-linear regression cannot be a good idea because of non-linear dependencies. On the other hand, b is a strong predictor of a and depends on a linearly. Therefore, I need to ensemble both linear regression and other methods (e.g., random forest) in a single model to have advantages of both.

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ Since you need to map linear and non linear features to the target variable, I think you could try using a simple neural network to model this kind of a relationship. $\endgroup$
    – Ankita Talwar
    Sep 23, 2020 at 16:32
  • 1
    $\begingroup$ If some features cause non-linear dependence, maybe polynomial regression? Or some feature engineering to make relationship more linear. $\endgroup$
    – kate-melnykova
    Sep 23, 2020 at 18:03
  • 1
    $\begingroup$ why a is in both place ? feature and response ? $\endgroup$
    – user702846
    Sep 24, 2020 at 12:29
  • $\begingroup$ @user702846, It's just response (target). which sentence has confused you? $\endgroup$
    – Amir Ch
    Sep 24, 2020 at 14:52
  • $\begingroup$ @AmirCh so the response (target) is also a feature ? both are noted with 'a' $\endgroup$
    – user702846
    Sep 25, 2020 at 8:49

1 Answer 1

Reset to default
0
$\begingroup$

The simple answer is to experiment.

You did a quite detailed analysis of the relations between your features and response variable and that's definitely a good idea, but don't be afraid to experiment with various models, even those which don't seem perfectly adapted to the task. Why? Because the one thing that such an analysis by individual features doesn't show is the patterns which happen with combinations of features, and that's what most ML algorithms are usually good at finding. This is why one often finds surprisingly good results with methods expected to perform poorly.

So my advice is to start with a range of simple methods known to be quite robust, for instance Random Forest, SVM, logistic (or even linear) regression. A good strategy is to start with simple methods and then try to improve on that based on analyzing the results. For example a more advanced approach would be to use ensemble learning: train different types of models and then a meta-model which relies on their predictions.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.