If I use RandomizedSearchCV to find the optimal hyperparams of a model, can I create another model, with those parameters, to calibrate probabilities using CalibratedClassifierCV? The new model is not fitted, so I can use the training set to train the probability calibrator using the same folds as the ones in RandomizedSearchCV. My question basically is, are those parameters still valid for calibrating probabilities? Otherwise, how can I merge these two operations?

  • $\begingroup$ Not sure I fully understand the question, but you basically need a separate (or held out) dataset to do calibration properly. scikit-learn.org/stable/modules/calibration.html#calibration. $\endgroup$
    – njp
    Commented Apr 5, 2023 at 23:15
  • $\begingroup$ Check the methodology explained here: machinelearningmastery.com/… It says that it can be performed even on the training set if the model has not been fitted. So, my question is, can I perform it after having found the best hyperparams with RandomizedSearchCV? $\endgroup$ Commented Apr 6, 2023 at 7:35


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.