As I increase the number of trees in scikit learn's GradientBoostingRegressor, I get more negative predictions, even though there are no negative values in my training or testing set. I have about 10 features, most of which are binary.

Some of the parameters that I was tuning were:

  • the number of trees/iterations;
  • learning depth;
  • and learning rate.

The percentage of negative values seemed to max at ~2%. The learning depth of 1(stumps) seemed to have the largest % of negative values. This percentage also seemed to increase with more trees and a smaller learning rate. The dataset is from one of the kaggle playground competitions.

My code is something like:

from sklearn.ensemble import GradientBoostingRegressor

X_train, X_test, y_train, y_test = train_test_split(X, y)

reg = GradientBoostingRegressor(n_estimators=8000, max_depth=1, loss = 'ls', learning_rate = .01)

reg.fit(X_train, y_train)

ypred = reg.predict(X_test)

Remember that the GradientBoostingRegressor (assuming a squared error loss function) successively fits regression trees to the residuals of the previous stage. Now if the tree in stage i predicts a value larger than the target variable for a particular training example, the residual of stage i for that example is going to be negative, and so the regression tree at stage i+1 will face negative target values (which are the residuals from stage i). As the boosting algorithm adds up all these trees to make the final prediction, I believe this can explain why you may end up with negative predictions, even though all the target values in the training set were positive, especially as you mentioned that this happens more often when you increase the number of trees.

  • $\begingroup$ This is the correct answer. $\endgroup$ – hahdawg Jun 26 '18 at 17:38

In general regression models (any) can behave in an arbitrary way beyond the domain spanned by training samples. In particular, they are free to assume linearity of the modeled function, so if you for instance train a regression model with points:

X     Y
10    0
20    1
30    2

it is reasonable to build a model f(x) = x/10-1, which for x<10 returns negative values.

The same applies "in between" your data points, it is always possible that due to the assumed famility of functions (which can be modeled by particular method) you will get values "out of your training samples".

You can think about this in another way - "what is so special about negative values?", why do you find existance of negative values weird (if not provided in training set) while you don't get alarmed by existance of lets say... value 2131.23? Unless developed in such a way, no model will treat negative values "different" than positive ones. This is just a natural element of the real values which can be attained as any other value.

  • $\begingroup$ With regards to your set of questions, I think it is purely the negative values are easier to identify as anomalies because they have that "-" in front of them or clearly go below zero on graphs. The question could just as easily be "Why does Gradient Boosting regression predict previously unseen values?". Maybe you could try to expand on that? It would certainly get you an up vote from me. $\endgroup$ – josh Sep 16 '16 at 11:15
  • 1
    $\begingroup$ @lejlot -- Generally speaking, this is not true. Regression models with logistic, or tanh activations are often guaranteed to have outputs within some bounds. $\endgroup$ – user48956 Oct 4 '16 at 18:23
  • $\begingroup$ @user48956 answer states "can behave in arbitrary manner", I am not claiming that you cannot force some constraints, of course you can - answer only states that there is no "data dependent" constraint (unless you have very specific model which has this built in construction) - if you add this manually as an expert - it is up to you. $\endgroup$ – lejlot Oct 4 '16 at 18:50
  • 1
    $\begingroup$ Random forests and nearest-neighbor models cannot make predictions outside their training range, for example. $\endgroup$ – Ben Reiniger Jan 14 at 22:24

The default number of estimators is 100. Reducing the number of estimators may work.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.