3
$\begingroup$

I am trying recently to optimize models but for some reason, whenever I try to run the optimization the model score in the end is worse than before, so I believe I do something wrong.

in order to optimize my model I define param grid and than fit with the train data and then according to the results run again with nre parameters, e.g-

#ROUND 1
param_grid={
    'max_depth': [3,4,5],
    'learning_rate':[0.1,0.01,0.05],
    'gamma': [0,0.25,1.0],
    'reg_lambda':[0,1.0,10.0],
    'scale_pos_weight':[1,3,5]
}

grid_search = GridSearchCV(estimator = clf_xgb, param_grid = param_grid, 
                          cv = 3, n_jobs = -1, verbose = 2)
grid_search.fit(X_train,y_train)
grid_search.best_params_

>>>.....

(and now based on the result changing the params...)

after this step I choose the best hyperparameters and run the model;

clf_xgb=xgb.XGBClassifier(seed=42,
                         objective='binary:logistic',
                         gamma=0,
                         learn_rate=0.7,
                         max_depth=6,
                         reg_lambda=0.8,
                         scale_pos_weight=1,
                         subsample=0.9,
                         cilsample_bytree=0.5)

clf_xgb.fit(X_train,
           y_train,
           verbose=True,
           early_stopping_rounds=10,
           eval_metric='aucpr',
           eval_set=[(X_test,y_test)])

The problem is that when I check the model score

clf_xgb.score(X_test,y_test)

I always get lower score than what I got before the optimization which makes me suspect that I'm missing something in the way doing it/basic principle in this process.

Is it possible that after running the optimization my score won't get better (and even worse?) ? Where is my mistake? Are there other parameters that could influence or improve my model?

$\endgroup$
3
  • 1
    $\begingroup$ When you try your classifier the hyper-parameters do not match the range of the param-grid. For instance, the learning rate is 0.7 whereas in your grid search you have a list with values [0.1,0.01,0.05], why is that? $\endgroup$
    – 20roso
    Sep 21 '20 at 17:43
  • $\begingroup$ @Grzegors I did 0.7 because when I run it first round I have gotten 0.1 and then I tried bigger numbers and in the end it was 0.7 $\endgroup$
    – Reut
    Sep 22 '20 at 6:16
  • $\begingroup$ @Reut Could you describe what your process was for optimization? And more about your data. $\endgroup$
    – Oaxacamatt
    Sep 24 '20 at 15:41
6
+50
$\begingroup$

Is it possible that after running the optimization my score won't get better (and even worse?) ?

Yes, theoretically, by pure luck, it is possible that your initial guess, before optimization of hyper-parameters, provides better results than the best of parameter combination found in the parameters grid. However, assuming you have enough data and your parameter grid is wide enough it is rather unlikely that the tuning of hyper-parameters would not be able to find better results. Such behavior rather indicates that something is wrong with your approach or your data.

If understand correctly, the choice of the best parameters is based on the cv results on training data, while in your final run the performance is assessed based on test dataset. If the distribution of training and test data differ significantly it could lead to the situation when the parameters providing the best results on the training data perform poorly on test data.

Where is my mistake?

As already mentioned by others, the parameters you are testing after the tuning were not included in the parameter grid. In this case it is incorrect to talk about the model performance "after running the optimization".

I suggest the following in order to investigate and fix the problem

  • Instead of using the hard-coded parameters in the XGBClassifier call, use the optimal parameters found by tuning process, i.e. grid_search.best_params_. Furthermore, if you think that subsample and cilsample_bytree (a typo?) are relevant parameters include them in the parameters grid.
  • Increase the cv parameter to e.g. 5-10, the results with cv = 3 might be very unstable. You can assess the stability of your current results by using different random seeds and repeating the entire exercise.
  • Make sure that you use the consistent parameters in tuning process and in the final evaluation, or just include these parameters in the parameters grid if possible. In particular, check early_stopping_rounds and eval_metric.

Are there other parameters that could influence or improve my model?

  • From your code it is unclear how many rounds you use. Either increase n_estimators or include it in the parameters grid.
  • Given that you use AUCPR you might need to explicitly set the parameter maximize=True, otherwise in your final run you could minimize the AUCPR, which could explain poor results.
$\endgroup$
0
$\begingroup$

This question is a little wrong-worded. You cannot get worse after optimization, otherwise it wouldn't be optimization! (At worst you are at the same performance like before, getting the exact same parameters you already had)

As Grzegorz points out in a comment, first of all your parameter list isn't complete and doesn't contain the values you use later. For example the learning rate, but also max_depth. Secondly, a grid search where you don't really know where to look should contain a much larger variance for the parameters. You check [0.1, 0.01, 0.05] for the learning rate, but did you check [0.0001, 0.001, 1.]? The learning rate might be a bad example here but I hope it gets the point across, you might want to check magnitude/scale first, e.g. powers of ten, before checking small variations.

Depending on your dataset, the difference between runs with the same values might also come from different seeds! Check that you either always set the same seed, or try it enough times with different seeds to get a comparable answer (for example with KFold).

Is your model even converging for every training? Where do you make sure that you train long enough? You can plot the loss for the training and test sample and check if it's converging or not. This can be controlled with n_estimators in xgboost I believe.

$\endgroup$
4
  • 2
    $\begingroup$ @n-kiefer I find your first paragraph presumptuous. The question posed IS the question. How can one think the question is not valid because of wording. This commentary is not helpful to the investigator. I feel it stifles curiosity or expression. Smugness like this is disheartening to me. In limited cases it is possible to find poor testing values after optimization. One, the optimization was not complete but coded correctly, see above. Two, if the sample population is small then even after optimization it is possible to find poor results. $\endgroup$
    – Oaxacamatt
    Sep 24 '20 at 15:39
  • 1
    $\begingroup$ The answer was not intented to be smug, presumptuous or anything like that. It was simply a short take on the question: is it possible to get worse after optimization? I never said it was invalid, and even attempted to give more pointers as to what could be done. I even (like you) thought of the problem of small datasets and recommended a KFold strategy. The question can indeed be read as: why is the performance worse on the test set compared to the training set? But I try to answer what i think the question is as in the title. $\endgroup$
    – N. Kiefer
    Sep 24 '20 at 16:34
  • $\begingroup$ my apologies... $\endgroup$
    – Oaxacamatt
    Sep 24 '20 at 16:48
  • 1
    $\begingroup$ The tone of @N.Kiefer answer is a bit off, but he is mostly right, optimization performed in the example given in the question is not broad enough to search the hyperparameter space: optimization done correctly can't give worst results than arbitrary choice of hyperparameters, it will at least give the same result (or slightly different given random effects if a random seed is not set). $\endgroup$ Sep 25 '20 at 1:20
0
$\begingroup$

There is nothing wrong in your code or process. Often times in machine learning performance on the test dataset is lower than than performance on the training data set. Your model is not generalizing perfectly to the data it has not seen before (i.e., the test dataset).

$\endgroup$
-2
$\begingroup$

When you do hyper-parameter tuning, you improve the regularization of the model. Before you did optimization, you could be overfitting. After optimization you regularized your model and now it performs just right.

enter image description here

Then your model score will be worse after optimization on training set. Your model score will also be bad for test set if your model heavily relies on one feature for classification.

You can use learning curve to see how the curve changes when you use optimization vs when you don't. And you can use df.corr() to see the correlation matrix for the correlation between feature values and target values.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.