Can I use scikit learn's cross_val_predict with cross_validate?

Question

I am looking to make a visualization of my cross validation data in which I can visualize the predictions that occurred within the cross validation process. I am using scikit learn's cross_validate to get the results of my bayesian ridge model's (scikit learn BayesianRidge) performance, but am unsure if my plot using cross_val_predict expresses the same predictions? My plot is a one-to-one plot of the predicted labels that occurred during cross validation versus the observed labels the model trained on. I use the same number of folds in both cross_validate and cross_val_predict.

Basically, I just want to know if the plot I make with cross_val_predict can be described by the returned performance metrics from cross_validate?

Thanks for the help

Answer 1

No, the folds used will (almost surely) be different.

You can enforce the same folds by defining a CV Splitter object and passing it as the cv argument to both cross-validation functions:

cv = KFold(5, random_state=42)
cross_validate(model, X, y, cv=cv, ...)
cross_val_predict(model, X, y, cv=cv, ...)

That said, you're fitting and predicting the model on each fold twice by doing this. You could use return_estimator=True in cross_validate to retrieve the fitted models for each fold, or use the predictions from cross_val_predict to generate the scores manually. (Either way though, you'd need to use the splitter object to slice to the right fold, which might be a little finicky.)