I've got a problem with understanding the CV parameter in cross_validate. Could you check if I understand it correctly?
I'm running ML algorithms in big set of data (train 37M rows), therefore I would like to run a big validation procedure to choose the best model.
Using ShuffleSplit, I want to build 100 different ways of splitting data in random way:
cv_split = model_selection.ShuffleSplit(n_splits = 100, test_size = .1, train_size = .9, random_state = 0
Then I want to use it as CV hyperparameter in cross_validate:
cv_results = model_selection.cross_validate(model, X, Y, cv = cv_split)
Does it mean that my Train set (X & Y) is divided into 100 random samples (each is then divided into: train (90% of sample), test (10 % of sample)) and during cross_validation model is built for each sample separetly (fitted on 90% of particular one/10 sample and tested on remaining 10% if this sample) and the mean prediction of those 100 models is the result? Also, if I am using Shaffle, does it mean that particular row can be in multiple samples and other will not be in any of them?
In other words, 37M set is devided:
First Sample 370k XY1, 90% *3.7 = 333k rows as XY1_1(train), 37k as XY1_2 (test); model fitted on .fit(X1_1, Y1_1), predition is build on .predict(X1_2) and validated against Y1_2
Second Sample 370k, 333k rows as XY2_1 and 37k rows as XY2_2; model fitted on .fit(X2_1, Y2_1), prediction built on .predict(X2_1) and validated against Y2_2
I am not sure If the second explanation is more clear. But this is how I structure it in my head.
I also read scikit.learn guide: Cross-validation: evaluating estimator performance but I am still not sure