0
$\begingroup$

I have a classification problem where i am trying to predict if the data returns a 1 or 0. So you're classic binary classification. I have my set of data that I have split into the dependent variables (ones i'm training on) and the independent variable (my target that i'm predicting, either a 0 or 1). I am using log loss as the scoring metric for my model.

Firstly I am using the cv function in xgboost to figure out the number of estimators I need as it stops when the log loss hasn't improved over 50 rounds. I then train my model and predict. My code is below:

def modelfit(alg, dtrain, dtarget, useTrainCV=True, cv_folds=5, early_stopping_rounds=50):

    if useTrainCV:
        # gets the xgb parameters specifically.
        xgb_param = alg.get_xgb_params()

        # this is the internal xgb dataframe that is for efficiency. We map the training data to the labels.
        xgtrain = xgb.DMatrix(dtrain.values, label=dtarget)

        # this performs cross validation on the dataset. As our data is not really time dependent we can afford to cross
        # validate. It stops when it hasnt improved for 50 rounds. This is only for determining n_estimators
        cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=alg.get_params()['n_estimators'], nfold=cv_folds,
            metrics='logloss', early_stopping_rounds=early_stopping_rounds)

        print(f'Optimal n_estimators - {cvresult.shape[0]}')

        # this sets the most optimal n_estimators parameter into the booster.
        alg.set_params(n_estimators=cvresult.shape[0])

    # fit the algorithm on the data and set evaluation metric
    alg.fit(dtrain.values, dtarget, eval_metric='logloss', eval_set=[(dtrain.values, dtarget)])

    print(alg.evals_result())

    # predict training set:
    dtrain_predictions = alg.predict(dtrain.values)
    print(dtrain_predictions)
    dtrain_predprob = alg.predict_proba(dtrain.values)[:,1]

    # print model report:
    print("\nModel Report")
    print("Log Loss Score (Train): %f" % metrics.log_loss(dtarget, dtrain_predprob))

I then run this function on this particular XGBoostClassifier:

#Choose all predictors
xgb1 = XGBClassifier(
 learning_rate =0.1,
 n_estimators=1000,
 max_depth=5,
 min_child_weight=1,
 gamma=0,
 subsample=0.8,
 colsample_bytree=0.8,
 objective= 'binary:logistic',
 scale_pos_weight=1, 
 nthread=-1,
 seed=27)

modelfit(xgb1, X, y)

The logloss value that is returned is: 0.577496 and number of estimators is 65.

I then turn to GridSearchCV to tune the other parameters and I start with:

param_test1 = {
 'max_depth' : range(1,10),
 'min_child_weight' : range(1,6)
}

Note how the original max depth and min child weight are contained within these ranges that i used in xgb1 classifier.

xgb2 = XGBClassifier(
        learning_rate =0.1, 
        n_estimators=65,
        max_depth=5,
        min_child_weight=1, 
        gamma=0, 
        subsample=0.8, 
        colsample_bytree=0.8,
        objective= 'binary:logistic',
        nthread=-1,
        scale_pos_weight=1, 
        seed=27
)

gsearch1 = GridSearchCV(
    estimator = xgb2, 
    param_grid = param_test1, scoring='neg_log_loss', n_jobs=-1, cv=5
)

gsearch1.fit(X, y)
gsearch1.best_params_, gsearch1.best_score_

However this returns me with:

(
{'max_depth': 1, 'min_child_weight': 1}, -0.6275341839742403
)

So my question is how has the grid search said the best parameters are max_depth = 1 and min_child_weight = 1 and the log loss is 0.628 when previously before using GridSearchCV my model returned a better log loss of 0.577 with max_depth = 5 and min_child_weight = 1?

Any help would be appreciated please. Thanks!

$\endgroup$
0
$\begingroup$

Your modelfit prints the training score, but GridSearchCV bases its decisions on the out-of-fold average (and in particular best_score_ is an out-of-fold average score). This is an unfair comparison, and in particular your 0.577 is probably quite optimistically biased.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Interesting - but I am using 'neg_log_loss' as the scoring method in GridSearchCV. So how do I compare the best result from GridSearchCV to the result I get from modelfit ? $\endgroup$ – Sean O'Connor Jun 15 at 15:12
  • $\begingroup$ Ideally, get a cross-validation score in modelfit instead of the training score. If you'll always be using useTrainCV=True, the array cvresult has what you need. $\endgroup$ – Ben Reiniger Jun 15 at 15:35
  • $\begingroup$ But what I'm struggling to understand is how comes when I use xgb.cv() I get a logloss score of 0.577. But then when I try to use GridSearchCV how can I tell it to find the parameters that will optimise my logloss score the most? As when I do it and insert the parameters it found - it gives a worse score than my original? $\endgroup$ – Sean O'Connor Jun 15 at 16:02
  • $\begingroup$ @SeanO'Connor, the first and third sentences of your comment are addressed in my answer. For the second, you are already doing it correctly. Just note that GridSearchCV is optimizing for the test-fold score, not train-folds (as it should!). $\endgroup$ – Ben Reiniger Jun 15 at 17:36
  • $\begingroup$ Ah okay understood. So how would I make it optimise for the train folds instead of the test folds? $\endgroup$ – Sean O'Connor Jun 16 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.