0
$\begingroup$

I am looking to make a visualization of my cross validation data in which I can visualize the predictions that occurred within the cross validation process. I am using scikit learn's cross_validate to get the results of my bayesian ridge model's (scikit learn BayesianRidge) performance, but am unsure if my plot using cross_val_predict expresses the same predictions? My plot is a one-to-one plot of the predicted labels that occurred during cross validation versus the observed labels the model trained on. I use the same number of folds in both cross_validate and cross_val_predict.

Basically, I just want to know if the plot I make with cross_val_predict can be described by the returned performance metrics from cross_validate?

Thanks for the help

$\endgroup$

1 Answer 1

1
$\begingroup$

No, the folds used will (almost surely) be different.

You can enforce the same folds by defining a CV Splitter object and passing it as the cv argument to both cross-validation functions:

cv = KFold(5, random_state=42)
cross_validate(model, X, y, cv=cv, ...)
cross_val_predict(model, X, y, cv=cv, ...)

That said, you're fitting and predicting the model on each fold twice by doing this. You could use return_estimator=True in cross_validate to retrieve the fitted models for each fold, or use the predictions from cross_val_predict to generate the scores manually. (Either way though, you'd need to use the splitter object to slice to the right fold, which might be a little finicky.)

$\endgroup$
3
  • $\begingroup$ Great this is perfect, thank you! Just to make sure, is the repeated k fold splitter: RepeatedKFold(n_splits=3, n_repeats=50, random_state=123) also an acceptable splitter? Because the documentation says that there is a "different randomization in each repetition", but I am not sure if the random state standardizes that...? $\endgroup$ Nov 17, 2022 at 0:42
  • $\begingroup$ @lambdaChops Yes: each repetition's randomization is obtained from a numpy RandomState object, whose evolution is deterministic. $\endgroup$
    – Ben Reiniger
    Nov 17, 2022 at 1:45
  • $\begingroup$ Awesome, thanks! $\endgroup$ Nov 17, 2022 at 2:06

This site is temporarily in read-only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .