-1
$\begingroup$

I've got a problem with understanding the CV parameter in cross_validate. Could you check if I understand it correctly?

I'm running ML algorithms in big set of data (train 37M rows), therefore I would like to run a big validation procedure to choose the best model.

Using ShuffleSplit, I want to build 100 different ways of splitting data in random way:

cv_split = model_selection.ShuffleSplit(n_splits = 100, test_size = .1, train_size = .9, random_state = 0

Then I want to use it as CV hyperparameter in cross_validate:

cv_results = model_selection.cross_validate(model, X, Y, cv  = cv_split)

Does it mean that my Train set (X & Y) is divided into 100 random samples (each is then divided into: train (90% of sample), test (10 % of sample)) and during cross_validation model is built for each sample separetly (fitted on 90% of particular one/10 sample and tested on remaining 10% if this sample) and the mean prediction of those 100 models is the result? Also, if I am using Shaffle, does it mean that particular row can be in multiple samples and other will not be in any of them?

In other words, 37M set is devided:

  • First Sample 370k XY1, 90% *3.7 = 333k rows as XY1_1(train), 37k as XY1_2 (test); model fitted on .fit(X1_1, Y1_1), predition is build on .predict(X1_2) and validated against Y1_2

  • Second Sample 370k, 333k rows as XY2_1 and 37k rows as XY2_2; model fitted on .fit(X2_1, Y2_1), prediction built on .predict(X2_1) and validated against Y2_2

  • etc

I am not sure If the second explanation is more clear. But this is how I structure it in my head.

I also read scikit.learn guide: Cross-validation: evaluating estimator performance but I am still not sure

$\endgroup$
2
$\begingroup$

I'll answer this first:

if I am using Shaffle, does it mean that particular row can be in multiple samples and other will not be in any of them

If I've understood the question correctly, then yes, that is possible. See below: from sklearn.model_selection import ShuffleSplit

import numpy as np
from sklearn.model_selection import ShuffleSplit
X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
y = np.array([1, 2, 1, 2])
rs = ShuffleSplit(n_splits=3, test_size=.25, random_state=8)
rs.get_n_splits(X)
train = []
for train_index, test_index in rs.split(X):
    print("TRAIN:", train_index, "TEST:", test_index)

Output:

TRAIN: [1 0 3] TEST: [2]
TRAIN: [0 3 1] TEST: [2]
TRAIN: [1 3 0] TEST: [2]

Row 2 doesn't appear in any of the training sets.

Next, to answer your first question:

the mean prediction of those 100 models is the result

Not quite. From the documentation (which you linked to):

Returns: scores : dict of float arrays of shape=(n_splits,) Array of scores of the estimator for each run of the cross validation.

Let's see this in action, on the above example:

from sklearn linear_model
from sklearn.model_selection import cross_validate
lasso = linear_model.Lasso()
cross_validate(lasso, X, y, cv = rs)['test_score']

returns

Out[36]: array([ 0.,  0.,  0.])

So you see, it returns an array with the score on each cross-validation fold.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ This is a great explanation and exactly what I was looking for. Thanks! $\endgroup$ – Mateusz Konopelski May 15 '18 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.