I wish to train some data using the the Gradient Boosting Regressor of Scikit-Learn.
My questions are:
1) Is the algorithm able to capture non-linear relationships? For example, in the case of y=x^2, y increases as x approaches negative infinity and positive infinity. What if the graph looks like y=sin(x)?
2) Is the algorithm able to detect interactions/relationships among the features? Specifically, should I add features that are the sums/differences of the raw features to the training set?
GB method works by minimizing a loss function and by splitting each node in a fashion that produces high pure leaves. there is no population formula being estimated and therefore you can estimate all types of relations between the target and the features.
However I wouldn't put in the model correlated variables as:
For gradient boosted trees, there's generally no strong need to check for multicollinearity because of its robustness. But practically speaking, you still should do some basic checks. For example, if you discover that two variables are 100% the same, then of course there's no point in keeping both. Even if it's 98% correlated, it's usually okay to drop one variable without degrading the overall model.
Source: Quora
