5
$\begingroup$

I have a regression problem where I need to predict three dependent variables ($y$) based on a set of independent variables ($x$): $$ (y_1,y_2,y_3) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \beta_n x_n +u. $$

To solve this problem, I would prefer to use tree-based models (i.e. gradient boosting or random forest), since the independent variables ($x$) are correlated and the problem is non-linear with ex-ante unknown parameterization.

I know that I could use sklearn's MultiOutputRegressor() or RegressorChain() as a meta-estimator.

However, there is an additional twist to my problem, namely that I do know that $y_1 + y_2 - y_3 = x_1$.

In other words, there is a fixed relation between the three $y$ and one of the independent variables. So essentially, the value of $x_1$ needs to be "distributed" in a first best manner to the (unknow) targets $(y_1,y_2,y_3)$ for each observation, contingent on the remaining independent variables $x_2,\dots,x_n$.

Of course a naive approach would be, to squeze the predicted values together somehow, so to satisfy $\hat{y_1} + \hat{y_2} - \hat{y_3} = x_1$. However, I wonder if there are any other options to introduce a "hard constraint" such as $\hat{y_1} + \hat{y_2} - \hat{y_3} = x_1$ to some (tree-based) estimator.

I noticed this post. However, I would prefer a tree-based method for reasons mentionned above.

Improve this question
$\endgroup$
4
  • $\begingroup$ Is the relationship exact in the training data, or noisy? $\endgroup$
    – Ben Reiniger ♦
    Aug 24, 2021 at 13:32
  • $\begingroup$ The relation $y_1+y_2-y_3=x_1$ is almost exact (few minor residual values), while the effect of the remaining $x$ on $y$ in the sense of $ y_1,y_2,y_3(x_2,...,x_n)$ is rather noisy. $\endgroup$
    – Peter
    Aug 24, 2021 at 14:33
  • $\begingroup$ I'd need to think through it some more before upgrading this to an answer, but some things to think about: (1) model just $y_1, y_2$ and then predict $\hat{y}_3 = \hat{y}_1 + \hat{y}_2 - x_1$. (2) RegressorChain, with the $y$s in order, will do essentially that but with some flexibility to change the $\hat{y}_3$. (3) Trees already do multi-output regression in a single tree, so MultiOutputRegressor shouldn't be needed. (4) If the relationship were just in the $y$s, that would be captured automatically by trees, since the leaf values are averages and the relationship is linear. $\endgroup$
    – Ben Reiniger ♦
    Aug 25, 2021 at 14:40
  • 1
    $\begingroup$ Thanks for your comment: I'm currently using RegressorChain() and I wonder if I would benefit from using option (1) in a stacking process where each of the $y_i$ is determined as "residual" for parts of the data. So first stage: Estimate two of the $\hat{y}_i$ in a chain, determine the last $\hat{\hat{y}}_i$ as residual to $x_1$. Second stage: Use the $\hat{\hat{y}}_i$ in a further modeling step to see if this information helps to reduce MSE, MAE and ensure that $\hat{y}_1+\hat{y}_2-\hat{y}_3=x_1$ is met. Do you think something like this could work? $\endgroup$
    – Peter
    Aug 26, 2021 at 9:52

1 Answer 1

Reset to default
0
$\begingroup$

So there does not seem to be anything that is out of the box ready for this but, I found an example of someone doing something similar to what you want to do with a Random Forest. Here is the link: http://astrohackweek.org/blog/multi-output-random-forests.html

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for the hint. However, when I see this correctly, the approach is essentially a regression chain. I‘ll have a closer look but I guess it will not be possible to model a constraint like the one mentionned in my post. scikit-learn.org/stable/modules/generated/… $\endgroup$
    – Peter
    Aug 21, 2021 at 15:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.