0
$\begingroup$

I am looking to make a visualization of my cross validation data in which I can visualize the predictions that occurred within the cross validation process. I am using scikit learn's cross_validate to get the results of my bayesian ridge model's (scikit learn BayesianRidge) performance, but am unsure if my plot using cross_val_predict expresses the same predictions? My plot is a one-to-one plot of the predicted labels that occurred during cross validation versus the observed labels the model trained on. I use the same number of folds in both cross_validate and cross_val_predict.

Basically, I just want to know if the plot I make with cross_val_predict can be described by the returned performance metrics from cross_validate?

Thanks for the help

$\endgroup$

1 Answer 1

1
$\begingroup$

No, the folds used will (almost surely) be different.

You can enforce the same folds by defining a CV Splitter object and passing it as the cv argument to both cross-validation functions:

cv = KFold(5, random_state=42)
cross_validate(model, X, y, cv=cv, ...)
cross_val_predict(model, X, y, cv=cv, ...)

That said, you're fitting and predicting the model on each fold twice by doing this. You could use return_estimator=True in cross_validate to retrieve the fitted models for each fold, or use the predictions from cross_val_predict to generate the scores manually. (Either way though, you'd need to use the splitter object to slice to the right fold, which might be a little finicky.)

$\endgroup$
3
  • $\begingroup$ Great this is perfect, thank you! Just to make sure, is the repeated k fold splitter: RepeatedKFold(n_splits=3, n_repeats=50, random_state=123) also an acceptable splitter? Because the documentation says that there is a "different randomization in each repetition", but I am not sure if the random state standardizes that...? $\endgroup$ Nov 17 at 0:42
  • $\begingroup$ @lambdaChops Yes: each repetition's randomization is obtained from a numpy RandomState object, whose evolution is deterministic. $\endgroup$
    – Ben Reiniger
    Nov 17 at 1:45
  • $\begingroup$ Awesome, thanks! $\endgroup$ Nov 17 at 2:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.