# Different performance for splitting into test/train data vs. using cross-validation

I am training a linear model using the following scikit-learn setup:

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.svm import LinearSVC
from sklearn.metrics import f1_score
from sklearn.model_selection import cross_val_score

[...]

random_state=786543

max_iter=5, tol=None)
clf = LinearSVC(random_state=random_state, dual=True, C=1.5)

X_train, X_test, y_train, y_test, i_train, i_test = train_test_split(feature_matrix, y, indices, test_size=0.33, random_state=random_state)
clf.fit(X_train, y_train.values)
predicted_train = clf.predict(X_train)
predicted_test = clf.predict(X_test)
print('Train Accuracy: ' + str(np.mean(y_train == predicted_train)))
print('Test Accuracy: ' + str(np.mean(y_test == predicted_test)))
print('Test F1 micro: ' + str(f1_score(y_test, predicted_test, average='micro')))
print('Test F1 macro: ' + str(f1_score(y_test, predicted_test, average='macro')))
print('Test F1 weighted: ' + str(f1_score(y_test, predicted_test, average='weighted')))


Train Accuracy: 0.985129495926343

Test Accuracy: 0.9601936525013448

Test F1 micro: 0.9601936525013448

Test F1 macro: 0.9000889214688401

Test F1 weighted: 0.9590331562500389

But now I run

scores = cross_val_score(clf, feature_matrix, y, cv=5, scoring='f1_macro')
print(scores)


array([0.65860981, 0.84306338, 0.82113645, 0.83414211, 0.64665942])

How can this discrepancy be explained? I tested this using different random states.

A couple of points to consider:

• I am having multible classes (but only one label per sample)
• The dataset is skewed (so there are classes with many samples and some with very few classes)
• I have 45066 samples, 5222 features, 259 classes

The number of samples per class is:

sorted(list(np.unique(y, return_counts=True)[1]))


[1, 1, 1, 1, 1, 1, 1, 2, 3, 4, 4, 4, 4, 4, 7, 7, 8, 9, 9, 10, 10, 10, 10, 10, 10, 10, 10, 10, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 13, 13, 13, 13, 13, 13, 14, 14, 14, 14, 14, 14, 15, 15, 15, 15, 15, 16, 16, 16, 16, 16, 17, 17, 17, 17, 17, 18, 18, 18, 19, 19, 19, 19, 19, 19, 20, 20, 20, 20, 20, 20, 21, 21, 21, 22, 22, 22, 22, 23, 23, 24, 24, 24, 25, 25, 25, 26, 26, 27, 27, 27, 27, 27, 29, 29, 29, 29, 30, 30, 30, 30, 32, 32, 32, 34, 34, 35, 35, 35, 36, 36, 36, 36, 36, 36, 37, 37, 37, 37, 38, 38, 38, 38, 40, 40, 40, 41, 41, 45, 45, 45, 46, 46, 46, 46, 47, 47, 47, 48, 49, 50, 50, 52, 55, 56, 59, 59, 60, 60, 61, 61, 61, 65, 65, 67, 67, 69, 72, 73, 74, 75, 77, 77, 79, 80, 84, 85, 87, 93, 96, 97, 97, 103, 110, 112, 117, 123, 130, 139, 139, 141, 143, 146, 146, 147, 147, 150, 159, 161, 169, 170, 177, 180, 180, 189, 191, 196, 198, 199, 201, 202, 203, 203, 208, 211, 230, 236, 249, 255, 264, 268, 269, 300, 332, 347, 356, 358, 364, 388, 433, 469, 476, 484, 548, 652, 698, 723, 748, 753, 807, 815, 1013, 1200, 1222, 1243, 1274, 1447, 1643, 1741, 2900, 3909, 4627]

• One common problem is that samples aren't shuffled before splitting; check that? (And not related to the question, but those classes with so few samples would really worry me...) Feb 10 '20 at 13:29
• @BenReiniger: Good thoughts. scikit-learn.org/stable/modules/generated/… states that the data is shuffled by default. The dataset is just what I am given and I have no influence over its skewness. I agree it is worrysome, but isn't it a common problem? However, I am not sure what to do about it, since I am more involved in unsupervised learning usually. Any tipps? Feb 11 '20 at 10:29
• Using class_weight='balanced' in LinearSVC did not make a difference. Feb 11 '20 at 10:41
• @BenReiniger: I guess I solved it. See my solution. Feb 11 '20 at 13:02

# Reason for the discrepancy

Two aspects have to be considered regarding the split:

1. Is the split done in a stratified manner? (it should)
2. Is the data shuffled? (it should)

The line

X_train, X_test, y_train, y_test, i_train, i_test = train_test_split(feature_matrix, y, indices, test_size=0.33, random_state=random_state)


splits the data in a stratified manner by default (see parameter stratify) and it does shuffle by default (see parameter shuffle):

The line

scores = cross_val_score(clf, feature_matrix, y, cv=5, scoring='f1_macro')


also splits the data in a stratified manner (see parameter cv:

For integer/None inputs, if the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used. In all other cases, KFold is used.

), but it does not shuffle. This causes the bad results for this line.

# Solution

## Option 1: Shuffle the data beforehand:

import sklearn
scores = cross_val_score(clf, *sklearn.utils.shuffle(feature_matrix, df.eClass, random_state=42), cv=5, scoring='f1_macro')


## Option 2: use appropriate cross-validaton object

I also looked into using an appropriate cross-validation sheme by using a different object:

import sklearn
skf = sklearn.model_selection.StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
for train_index, test_index in skf.split(X, y):
print("TRAIN:", train_index, "TEST:", test_index)
X_train, X_test = X[train_index], X[test_index]
y_train, y_test = y.iloc[train_index], y.iloc[test_index]

clf.fit(X_train, y_train.values)
print('--------------------------------------')
predicted_train = clf.predict(X_train)
predicted_test = clf.predict(X_test)
print('Train Accuracy: ' + str(np.mean(y_train == predicted_train)))
print('Test Accuracy: ' + str(np.mean(y_test == predicted_test)))
print('Test F1 micro: ' + str(f1_score(y_test, predicted_test, average='micro')))
print('Test F1 macro: ' + str(f1_score(y_test, predicted_test, average='macro')))
print('Test F1 weighted: ' + str(f1_score(y_test, predicted_test, average='weighted')))
print('--------------------------------------')



see:

1. I would remove the classes with very few samples, as they create discrepancies in the model and also help with skew.
2. I would try to create new features by combining the features which are similar/ or have similar influence on the outcome. This is because as compared to the number of samples, you have too many features.
3. Try using logistic regression and see the results. Logreg works great with such data.
• Thank you for the tips! Wouldn't methods like SVM or Logistic Regression takes take care of the redundant features anyway? Afaik, tree-based methods would, as they would just ignore features that are not contributing any (additional, compared to other features) class-discriminating information. Feb 11 '20 at 16:45
• @Make42 why did you post a question!! You are already so well aware!! Feb 11 '20 at 17:36
• I am confused, what you mean. My original question was about the discrepancy regarding the results of the two experiments. You answered this with your first point. As far as I understand your point 2 and 3. are aimed to help me improve my results - which I appreciate. Did I get this right so far? Since I am not an expert in classification, I was wondering whether SVM / logistic regression take care of redundant features. I do not know whether this is the case - I am asking. So I am not sure what I am well aware of. Maybe my English is too bad. Feb 11 '20 at 17:41
• I am also not sure about my understand regarding the tree-based methods. If you know more than I, I am happy to learn. Feb 11 '20 at 17:44