I'm not quite sure how I should go about tuning xgboost before I use it as a meta-learner in ensemble learning.

Should I include the prediction matrix (ie. df containing columns of prediction results from the various base learners) or should I just include the original features?

I have tried both methods with just the 'n_estimators' tuned with F1 score as the metric for cross-validation. (learning rate =0.1)

Method 1: With pred matrix + original features:

n_estimators = 1 (this means only one tree is included in the model, is this abnormal? )
F1 Score (Train): 0.907975 (suggest overfitting)

Method 2: With original features only:

n_estimators = 1
F1 Score (Train): 0.39

I am getting rather different results for both methods, which makes sense as the feature importance plot for Method 1 shows that one of the first-level predictions is the most important.

I think that the first-level predictions by the base-learners should be included in the gridsearch. Any thoughts?

  • $\begingroup$ How does your base learner's scores (train and test) compare with the boosted score ? $\endgroup$ – VanillaSpinIce May 26 '18 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.