1
$\begingroup$
import pandas as pd
from sklearn.datasets import load_iris
from sklearn.svm import SVC

#import data
iris = load_iris()
X = pd.DataFrame(iris.data, columns=iris.feature_names) 
y = iris.target

#split data
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y,
 test_size = 0.2, shuffle=False)

#modeling using SVM
model = SVC(kernel='linear', C = 1)
model.fit(X_train, y_train)
model.score(X_test, y_test)

# here I get score: 0.8666666

# Now I use cross_validation with cv = 5,

from sklearn.model_selection import cross_val_score
model = SVC(kernel='linear', C = 1)
scores = cross_val_score(model, X, y, cv = 5, scoring = "accuracy")
scores

#here I got array([0.96666667, 1.        , 0.96666667, 0.96666667, 1.        ])

None of the numbers in the above arrays equal to 0.866666, I wonder why the inconsistency happens(since "cv = 5" matches the condition "test_size = 0.2").

$\endgroup$
1
$\begingroup$

One reason might be your train_test_split() setup. Using shuffle=False means you just use the first 80% examples of your data to train with no randomness. Have a look at your labels:

>>> y
Out: 
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

As you can see the iris dataset is sorted by its classes from 0 to 2. This means your train_test_split() will be unbalanced since class 2 will be underrepresented:

>>> y_train
Out: 
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
       2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

And even worse your test data only has one class:

>>> y_test
Out: 
array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
       2, 2, 2, 2, 2, 2, 2, 2])

On the other hand cross-validation will use a stratified split preserving the class distribution for all folds as described in the sklearn user guide:

For integer/None inputs, if the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used. In all other cases, KFold is used.

Which I'd expect to be leading to better results. Now, also giving the train_test_split() this advantage of balanced classes (stratified labels) gives me a different result:

>>> X_train, X_test, y_train, y_test = train_test_split(X, y,
 test_size = 0.2, stratify=y)

runfile(...)

>>> model.score(X_test, y_test)
Out: 1.0

$\endgroup$
0
$\begingroup$

The inconsistency simply happens because cross_val_score may compute different train/test splits than train_test_split, especially since you use shuffle=False.

Now, whether or not they should is a different question. To ensure that the splits are the same, first of all, I would use a random state to control the randomness. Second, cross_val_score has the cv parameter which allows you to define the splits yourself

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.