1
$\begingroup$

I tested my CatBoostModel model on part of data and get 0.92 score, but Kaggle public score was 0.9. I found new hyperparameters via randomsearch, new model score was 0.925, but on Kaggle score fell to 0.88.

What should I do to validate the model correctly?

$\endgroup$

1 Answer 1

2
$\begingroup$

In general, you should expect to get lower scores on test sets than validation sets, since you took advantage of validation data to tune your model. But for a correctly trained model, the difference between the validation and test sets must be small, as in 0.92 vs 0.9. To be more confident about your model's output, you can perform Cross-Validation.

Also, apparently, your model overfitted the training data after hyperparameter optimization. You can use regularization or early-stopping to prevent that.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.