I have a question about cross-validation using sklearn in Python (2.7).
I have updated this to include the code I use prior to cross-validation.
I import a csv into a dataframe. Some of these columns contain categories. So I use get_dummies to convert them to binary columns (i.e., One-Hot Encoding). This retains the non-categorical, removes the categorical data, and substitutes the categorical data with binary columns.
feature_df = pd.get_dummies(df, columns=feature_cat)
X = feature_df.drop(target_cat, axis=1)
print X # visually confirm it is formatted correctly
y = df[target_cat]# target_cat is a string containing the name of the column containing my target data. I have printed this out to confirm it is correctly extracting this from the df.
X = scale(X) # Scaling is important for preventing misleading results
X = pd.DataFrame(X)
(I also printed this out to confirm it contains and is formatted as it is supposed to be.)
# Note: sklearn LeaveOneOut() cross-validation needs X to be a data frame, although right now I am only using k-fold.
Then I create my pipeline:
model = DecisionTreeClassifier()
pipe = Pipeline([('pca', PCA()), ('clf', model)])
pipe.fit(X,y)
After this:
# Make predictions on training set:
predictions = pipe.predict(X)# Print initial accuracy
accuracy = metrics.accuracy_score(y, predictions)
print "Accuracy : %s" % "{0:.3%}".format(accuracy)kfold=KFold(n_splits=100)
acc_score = np.mean(cross_val_score(pipe, X, y, scoring='accuracy', cv=kfold))
print acc_score
When I measure my classifier accuracy using accuracy_score I get:
Accuracy :
88.416%
And my normalized confusion matrix is:
[[ 0.97 0. 0.02 0. 0.01 0.01]
[ 0.08 0.89 0.01 0. 0.01 0.01]
[ 0.09 0.01 0.88 0. 0.01 0.01]
[ 0.06 0.02 0.04 0.84 0.01 0.02]
[ 0.06 0.02 0.04 0.02 0.83 0.02]
[ 0.06 0.01 0.04 0.02 0.04 0.83]]
When I run cross validation using:
kfold=KFold(n_splits=10)
cross_val_score(pipe, X, y, scoring='accuracy', cv=kfold)(pipe is PCA and DecisionTreeClassification pipeline.)
It gives me a mean accuracy of:
0.231595981914
With the accuracies for each of the 10 folds:
[ 0.26901521 0.20176141 0.22177742 0.29967949 0.22435897 0.24919872 0.27163462 0.21153846 0.21794872 0.20192308]
While I would expect variation in the accuracy of the classifier, I wouldn't expect it all to be this completely different from the original accuracy, and I certainly wouldn't expect it to be consistently this different.
To further test the feasibility of the these results, I shuffled the rows of my original data frame containing both the features and targets. I ran it again, and now average cross-validation accuracy is:
0.553633419556
with individual fold accuracies of:
[ 0.55644516 0.56044836 0.53242594 0.55849359 0.53926282 0.54326923 0.55128205 0.55528846 0.57532051 0.54887821]
I try this a few more times by changing the seed for random and/or shuffling the shuffled dataset. My cross-validation accuracy remained pretty much the same. For example:
0.553075665661
Fold accuracies:
[ 0.5236189 0.54923939 0.54043235 0.55769231 0.55048077 0.55208333 0.54567308 0.55769231 0.56570513 0.58092949]
So then I increased the folds to 100, and the mean accuracy was still 0.55, and the 100 resulting accuracies from each test run were all between .49 (min) and .64(max).
If I run it with 100 folds on my original, non-shuffled dataset, I get a mean cross-validation accuracy score of 0.26, with individual accuracies ranging between .09 and .52, with most being .2 or .3.
Is it me or is this weird? Shouldn't these results be more consistent? Is there something wrong with metrics.accuracy_score (which I use to compute my original classification accuracy, prior to cross-validation)? Presumably this is also what cross_val_score uses when I set scorer to accuracy...
Any advice and/or feedback would be greatly appreciated!!! For what it's worth, I am using PyCharm on a Mac (Sierra). Thank you!
