inconsistency occurred when using cross_val_score in python

Question

import pandas as pd
from sklearn.datasets import load_iris
from sklearn.svm import SVC

#import data
iris = load_iris()
X = pd.DataFrame(iris.data, columns=iris.feature_names) 
y = iris.target

#split data
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y,
 test_size = 0.2, shuffle=False)

#modeling using SVM
model = SVC(kernel='linear', C = 1)
model.fit(X_train, y_train)
model.score(X_test, y_test)

# here I get score: 0.8666666

# Now I use cross_validation with cv = 5,

from sklearn.model_selection import cross_val_score
model = SVC(kernel='linear', C = 1)
scores = cross_val_score(model, X, y, cv = 5, scoring = "accuracy")
scores

#here I got array([0.96666667, 1.        , 0.96666667, 0.96666667, 1.        ])

None of the numbers in the above arrays equal to 0.866666, I wonder why the inconsistency happens(since "cv = 5" matches the condition "test_size = 0.2").

Answer 1

One reason might be your train_test_split() setup. Using shuffle=False means you just use the first 80% examples of your data to train with no randomness. Have a look at your labels:

>>> y
Out: 
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

As you can see the iris dataset is sorted by its classes from 0 to 2. This means your train_test_split() will be unbalanced since class 2 will be underrepresented:

>>> y_train
Out: 
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
       2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

And even worse your test data only has one class:

>>> y_test
Out: 
array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
       2, 2, 2, 2, 2, 2, 2, 2])

On the other hand cross-validation will use a stratified split preserving the class distribution for all folds as described in the sklearn user guide:

For integer/None inputs, if the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used. In all other cases, KFold is used.

Which I'd expect to be leading to better results. Now, also giving the train_test_split() this advantage of balanced classes (stratified labels) gives me a different result:

>>> X_train, X_test, y_train, y_test = train_test_split(X, y,
 test_size = 0.2, stratify=y)

runfile(...)

>>> model.score(X_test, y_test)
Out: 1.0

Answer 2

The inconsistency simply happens because cross_val_score may compute different train/test splits than train_test_split, especially since you use shuffle=False.

Now, whether or not they should is a different question. To ensure that the splits are the same, first of all, I would use a random state to control the randomness. Second, cross_val_score has the cv parameter which allows you to define the splits yourself