Different performance for splitting into test/train data vs. using cross-validation

Question

I am training a linear model using the following scikit-learn setup:

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.svm import LinearSVC
from sklearn.metrics import f1_score
from sklearn.model_selection import cross_val_score

[...]

random_state=786543

max_iter=5, tol=None)
clf = LinearSVC(random_state=random_state, dual=True, C=1.5)

X_train, X_test, y_train, y_test, i_train, i_test = train_test_split(feature_matrix, y, indices, test_size=0.33, random_state=random_state)
clf.fit(X_train, y_train.values)
predicted_train = clf.predict(X_train)
predicted_test = clf.predict(X_test)
print('Train Accuracy: ' + str(np.mean(y_train == predicted_train)))
print('Test Accuracy: ' + str(np.mean(y_test == predicted_test)))
print('Test F1 micro: ' + str(f1_score(y_test, predicted_test, average='micro')))
print('Test F1 macro: ' + str(f1_score(y_test, predicted_test, average='macro')))
print('Test F1 weighted: ' + str(f1_score(y_test, predicted_test, average='weighted')))

Train Accuracy: 0.985129495926343

Test Accuracy: 0.9601936525013448

Test F1 micro: 0.9601936525013448

Test F1 macro: 0.9000889214688401

Test F1 weighted: 0.9590331562500389

But now I run

scores = cross_val_score(clf, feature_matrix, y, cv=5, scoring='f1_macro')
print(scores)

array([0.65860981, 0.84306338, 0.82113645, 0.83414211, 0.64665942])

How can this discrepancy be explained? I tested this using different random states.

A couple of points to consider:

• I am having multible classes (but only one label per sample)
• The dataset is skewed (so there are classes with many samples and some with very few classes)
• I have 45066 samples, 5222 features, 259 classes

The number of samples per class is:

sorted(list(np.unique(y, return_counts=True)[1]))

[1, 1, 1, 1, 1, 1, 1, 2, 3, 4, 4, 4, 4, 4, 7, 7, 8, 9, 9, 10, 10, 10, 10, 10, 10, 10, 10, 10, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 13, 13, 13, 13, 13, 13, 14, 14, 14, 14, 14, 14, 15, 15, 15, 15, 15, 16, 16, 16, 16, 16, 17, 17, 17, 17, 17, 18, 18, 18, 19, 19, 19, 19, 19, 19, 20, 20, 20, 20, 20, 20, 21, 21, 21, 22, 22, 22, 22, 23, 23, 24, 24, 24, 25, 25, 25, 26, 26, 27, 27, 27, 27, 27, 29, 29, 29, 29, 30, 30, 30, 30, 32, 32, 32, 34, 34, 35, 35, 35, 36, 36, 36, 36, 36, 36, 37, 37, 37, 37, 38, 38, 38, 38, 40, 40, 40, 41, 41, 45, 45, 45, 46, 46, 46, 46, 47, 47, 47, 48, 49, 50, 50, 52, 55, 56, 59, 59, 60, 60, 61, 61, 61, 65, 65, 67, 67, 69, 72, 73, 74, 75, 77, 77, 79, 80, 84, 85, 87, 93, 96, 97, 97, 103, 110, 112, 117, 123, 130, 139, 139, 141, 143, 146, 146, 147, 147, 150, 159, 161, 169, 170, 177, 180, 180, 189, 191, 196, 198, 199, 201, 202, 203, 203, 208, 211, 230, 236, 249, 255, 264, 268, 269, 300, 332, 347, 356, 358, 364, 388, 433, 469, 476, 484, 548, 652, 698, 723, 748, 753, 807, 815, 1013, 1200, 1222, 1243, 1274, 1447, 1643, 1741, 2900, 3909, 4627]

Answer 1

Reason for the discrepancy

Two aspects have to be considered regarding the split:

1. Is the split done in a stratified manner? (it should)
2. Is the data shuffled? (it should)

The line

X_train, X_test, y_train, y_test, i_train, i_test = train_test_split(feature_matrix, y, indices, test_size=0.33, random_state=random_state)

splits the data in a stratified manner by default (see parameter stratify) and it does shuffle by default (see parameter shuffle):

see: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html

The line

scores = cross_val_score(clf, feature_matrix, y, cv=5, scoring='f1_macro')

also splits the data in a stratified manner (see parameter cv:

For integer/None inputs, if the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used. In all other cases, KFold is used.

), but it does not shuffle. This causes the bad results for this line.

see: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html

Solution

Option 1: Shuffle the data beforehand:

import sklearn
scores = cross_val_score(clf, *sklearn.utils.shuffle(feature_matrix, df.eClass, random_state=42), cv=5, scoring='f1_macro')

Option 2: use appropriate cross-validaton object

I also looked into using an appropriate cross-validation sheme by using a different object:

import sklearn
skf = sklearn.model_selection.StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
for train_index, test_index in skf.split(X, y):
    print("TRAIN:", train_index, "TEST:", test_index)
    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y.iloc[train_index], y.iloc[test_index]

    clf.fit(X_train, y_train.values)
    print('--------------------------------------')
    predicted_train = clf.predict(X_train)
    predicted_test = clf.predict(X_test)
    print('Train Accuracy: ' + str(np.mean(y_train == predicted_train)))
    print('Test Accuracy: ' + str(np.mean(y_test == predicted_test)))
    print('Test F1 micro: ' + str(f1_score(y_test, predicted_test, average='micro')))
    print('Test F1 macro: ' + str(f1_score(y_test, predicted_test, average='macro')))
    print('Test F1 weighted: ' + str(f1_score(y_test, predicted_test, average='weighted')))
    print('--------------------------------------')

see:

• https://scikit-learn.org/stable/modules/cross_validation.html
• https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html

Answer 2

1. I would remove the classes with very few samples, as they create discrepancies in the model and also help with skew.
2. I would try to create new features by combining the features which are similar/ or have similar influence on the outcome. This is because as compared to the number of samples, you have too many features.
3. Try using logistic regression and see the results. Logreg works great with such data.