1
$\begingroup$

I have one key relationship between a numeric independent variable X and a numeric dependent variable Y, which is like a negative exponential function determined by 2 parameters. There are other independent variables. One numeric, the rest are categorical.

At the moment I am using quantile regression forests. They work pretty well. However, the aforementioned negative exponential relationship is not guaranteed.

Is there a way to achieve the above. For example, could this be modeled like a hierarchical model. Here the parameters of negative exponential function over X are adjusted depending on the other features. In fact, could one model this as a ANN (e.g. using KERAS) where the negative exponential function is the output layer - similar to the softmax function?

Any feedback would be very much appreciated!

$\endgroup$
  • $\begingroup$ I believe, but I've not tried it yet, that both catboost and xgboost have monotonic constraints. i.e. xgboost.readthedocs.io/en/latest/tutorials/monotonic.html and github.com/catboost/catboost/tree/master/catboost/benchmarks/… $\endgroup$ – David Waterworth Aug 6 '19 at 6:19
  • $\begingroup$ Looks great thanks! Will have a look. $\endgroup$ – cs0815 Aug 6 '19 at 6:30
  • $\begingroup$ Could a "linear regression" be a solution? In this case you could specify parameterization of the estimation function. $\endgroup$ – Peter Aug 6 '19 at 10:22
  • $\begingroup$ @Peter - the relationship between the numeric independent variable X and a numeric dependent variable Y is non-linear - unless their is some transformation for a negative exponential function? $\endgroup$ – cs0815 Aug 6 '19 at 11:11
  • $\begingroup$ e.g. ln? Guess a linear hierarchical model could work ... $\endgroup$ – cs0815 Aug 6 '19 at 11:17
1
$\begingroup$

There are multiple algorithms that support monotonicity, i.e., can learn a monotonic decision rule.

It is easy to enforce a monotonicity constraint when using linear regression and logistic regression; you simply enforce that the corresponding coefficient for that variable is non-negative during optimization.

xgboost supports monotonicity constraints. However, beware that it is "best-effort" and does not guarantee that the model will be globally 100% monotonic for the entire space of data values.

Neural networks can support monotonicity constraints, by enforcing that the weights are non-negative.

I don't know how to enforce monotonicity with decision trees or random forest classifiers. See, e.g., https://cs.stackexchange.com/q/69220/755 for a counterexample showing that even if your training set is monotonic, the resulting decision tree might learn a non-monotonic rule.

Some references to check out: https://stats.stackexchange.com/q/257049/2921, https://stats.stackexchange.com/q/342651/2921, https://stats.stackexchange.com/q/341422/2921.

$\endgroup$
  • $\begingroup$ thanks. Please note that I also require to predict quantiles and this is a regression problem. Had some modest success with the qrnn package ... $\endgroup$ – cs0815 Sep 13 '19 at 7:21
  • $\begingroup$ @cs0815, Got it. All of the methods I describe have regression versions (linear regression obviously supports regression; gradient-boosted trees can be used for regression; neural networks can be used for regression, if you use an appropriate loss function). Perhaps you might like to write your own answer explaining how the qrnn package solves this problem? I'd be interested to learn about that, it sounds like it might be useful to others as well. $\endgroup$ – D.W. Sep 13 '19 at 18:01
  • $\begingroup$ Sorry I am not talking about standard regression, which predicts a point estimate, but about quantile regression. $\endgroup$ – cs0815 Sep 13 '19 at 18:57
0
$\begingroup$

This R package works for me:

https://cran.r-project.org/web/packages/Rborist/vignettes/rborist.html

The regMono option can be used. I also end up with quantile regression forests.

$\endgroup$
  • $\begingroup$ Actually it does not work 100%. Monotonicity is not ensured globally! $\endgroup$ – cs0815 Sep 13 '19 at 7:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.