K fold cross validation algorithm

Question

Suppose I want to apply cross validation without any inbuilt function. I have multi classifier problem which is perfectly divided in n section where n is the number of target features. If I divide my training data sets in to k folds sets then I might end up having single target feature in one set. So to solve this problem I shuffle my training set and target and I am able to achieve 0.96 accuracy compared to 0.98 when I use inbuilt method in sklearn. Is there a better approach to divide your data perfectly in to k folds?

Answer 1

It seems there is a small bit of ambiguity in the question. But my best guess is that you are looking for stratifiedkfold (Emre has mentioned that in a comment) or second best guess is labelkfold. See: http://scikit-learn.org/0.17/modules/generated/sklearn.cross_validation.StratifiedKFold.html#sklearn.cross_validation.StratifiedKFold and http://scikit-learn.org/0.17/modules/generated/sklearn.cross_validation.LabelKFold.html#sklearn.cross_validation.LabelKFold

It would help if you specify which inbuilt cross_validation in sklearn you tried, since there are several to choose from: http://scikit-learn.org/0.17/modules/classes.html#module-sklearn.cross_validation

Additionally, please also let us know about the number of data points available, since that will contribute to deciding on the numeric value of k. The point being that you don't want to bias your training due to an counter-productively high value of k for the available n data points. Herein lies the reason for the labelkfold approach since it will ensure that the same data point does not repeatedly occur in multiple folds, which would end up biasing your classifier, and we want to avoid this.

Finally, if your training time is of the order of a few minutes, you could try several cross_validation schemes. It will be a good learning experience and might provide some useful insights into the data :-)

-- Since, you are exploring the Iris dataset, I've gone ahead and created a small demo of the effect of different cross validation methods on classifier evaluation. I've put in a small range for relevant parameters for each of the methods considered.

I've included my code for you to experiment with, you can try more cross-validation methods. Iris data set has small set of instances and there isn't too much 'craziness' in the distribution of the classes. If you tried it on some real-world data with a lot of bias and noise, you might see a more pronounced effect of cross-validation (use error-bars).

--

    """
Evaluate cross-validation schemes for Iris dataset
"""

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn import svm
from sklearn import datasets
from sklearn.model_selection import train_test_split
from time import time
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import RepeatedStratifiedKFold

iris = datasets.load_iris()
X, y = iris.data, iris.target

""" visually inspect iris dataset """
iris_sns = sns.load_dataset('iris')
g = sns.pairplot(iris_sns, hue='species', markers=["o", "s", "D"])
plt.savefig(filename='iris_pairplot.png')
""" Dimensions of data matrix and target labels """
print(X.shape, y.shape)

"""
cross-validation: the apparent silver bullet to over-fitting
The available data and label matrices are split into set of training and validation matrices
We first begin with mere evaluation of classifier trained on different random sets 
of training and validation data sampled from the entire available data.
By averaging the computed classification performance on each training subset,
we will acquire a statistically reliable estimate of how this classifier might perform on unseen test data.
Subsequently, we will train classifier on multiple subsets of training data to actually improve the
learnt classifier itself.
"""

""" The effect of train-test split sizes"""
test_sizes = [0.1 * i for i in range(1, 10)]
scores = []
for test_size in test_sizes:
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_size, random_state=round(time()))
    print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)
    clf = svm.SVC(kernel='linear', C=0.1).fit(X_train, y_train)
    scores.append(clf.score(X_test, y_test))

fig1 = plt.figure(1)
plt.plot(test_sizes, scores, 'k-d')
plt.title('Iris dataset, effect of proportion of train-test splits')
plt.xlabel('proportion of test-images')
plt.ylabel('classification score')
plt.savefig('train-test-splits.png')
""" The effect of number of cross-validations """
cvs = [i for i in range(2, 11)]
cv_scores_means = []
cv_scores_std = []
for cv in cvs:
    clf = svm.SVC(kernel='linear', C=0.1, random_state=round(time()))
    scores = cross_val_score(clf, X, y, cv=cv)
    cv_scores_means.append(scores.mean())
    cv_scores_std.append(scores.std())

fig2 = plt.figure(2)
plt.errorbar(cvs, cv_scores_means, yerr=cv_scores_std, fmt='k-o')
plt.title('Effect of number of cross-validations')
plt.xlabel('number of cross-validation')
plt.ylabel('mean classification score')
plt.savefig('number-of-cv.png')
""" The effect of k-stratified splits"""
num_splits = [i for i in range(2,11)]
scores_splits = []
for n_splits in num_splits:
    skf = StratifiedKFold(n_splits=n_splits)
    clf = svm.SVC(kernel='linear', C=0.1)
    scores = []
    for train, test in skf.split(X, y):
        clf.fit(X[train], y[train])
        scores.append(clf.score(X[test], y[test]))
    scores_splits.append(np.average(scores))

fig3 = plt.figure(3)
plt.plot(num_splits, scores_splits, 'k-s')
plt.title('Effect of number of stratified splits')
plt.xlabel('number of splits')
plt.ylabel('mean classification score')
plt.savefig('num-stratified-splits.png')
""" the effect of k-label splits"""
num_folds = [i for i in range(2, 11)]
num_repeats = [i for i in range(2, 10, 2)]
line_colors = ['r', 'g', 'b', 'k', 'y']
scores_repeats = []
for n_repeat in num_repeats:
    scores_folds = []
    for n_fold in num_folds:
        rskf = RepeatedStratifiedKFold(n_splits=n_fold, n_repeats=n_repeat, random_state=round(time()))
        clf = svm.SVC(kernel='linear', C=0.1)
        scores = []
        for train, test in rskf.split(X, y):
            clf.fit(X[train], y[train])
            scores.append(clf.score(X[test], y[test]))
        scores_folds.append(np.average(scores))
    scores_repeats.append(scores_folds)
fig4 = plt.figure(4)
handles = []
for i in range(len(num_repeats)):
    h, = plt.plot(num_folds, scores_repeats[i], line_colors[i]+'-x', label="#rep: "+str(num_repeats[i]))
    handles.append(h)
plt.title('Effect of number of splits and repeats for repeated stratified k fold')
plt.xlabel('number of splits')
plt.ylabel('mean classification score')
plt.legend(handles=[h for h in handles])
plt.savefig('num-split-repeat-stratified.png')
plt.show()