# Variance in cross validation score / model selection

Between cross-validation runs of a xgboost classification model, I gather different validation scores. This is normal, the Train/validation split and model state are different each time.

flds = self.gsk.Splits(X, cv_folds=cv_folds)
cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=xgb_param['n_estimators'], nfold=cv_folds, flds, metrics='auc', early_stopping_rounds=50, verbose_eval=True)

self.model.set_params(n_estimators=cvresult.shape[0])


To make the parameters selection, I run multiple times this CV and average the results in order to attenuate those differences.

Once my model parameters have been "found", what is the correct way to train the model, which seems to have some inner random states ?

Do I :

• train on the full train set and hope for the best?
• keep the model with the best validation score in my CV loop (I am concerned this will overfit)?
• bag all of them?
• bag only the good ones?