2
$\begingroup$

How to use k-fold cross validation for MNIST dataset? I read article documentation on sci-kit learn ,in that example they used the whole iris dataset for cross validation.

from sklearn.model_selection import cross_val_score
clf = svm.SVC(kernel='linear', C=1)
scores = cross_val_score(clf, iris.data, iris.target, cv=5)
scores                                              

for example while importing mnist dataset in keras

from keras.datasets import mnist
(Xtrain,Ytrain),(Xtest,Ytest)=mnist_load()

in this dataset is already divided in test and train , so to apply cross validation on the entire dataset do we need to make Xtrain and Xtest as one entity to exploit the whole data.

$\endgroup$
6
  • $\begingroup$ Split your training set using sklearn train_test_split and making a validation set out of the training set itself or go for stratified sampling X_train, X_val, Y_train, Y_val = train_test_split(X_train, y_train, test_size = 0.1, random_state=42) $\endgroup$ – Aditya Mar 19 '18 at 6:31
  • $\begingroup$ I don't want to split the data it will be taken by the crossvalidation function. $\endgroup$ – Boris Mar 19 '18 at 6:36
  • $\begingroup$ Yes it will be automatically done for you inside $\endgroup$ – Aditya Mar 19 '18 at 6:38
  • $\begingroup$ So there is no need of train_test_split $\endgroup$ – Boris Mar 19 '18 at 6:39
  • 1
    $\begingroup$ You should not use the test set for cross validation. Use a part of the training set only. This ensures that you are not overfitting to the test data. $\endgroup$ – Jon Nordby Apr 14 '19 at 19:24
0
$\begingroup$
from sklearn.model_selection import cross_val_score
clf = svm.SVC(kernel='linear', C=1)
scores = cross_val_score(clf, iris.data, iris.target, cv=5)
scores

They are not using the whole data for cross validation as such ( it's just an illusion)

When the cv argument is an integer, cross_val_score uses the KFold or StratifiedKFold strategies by default, the latter being used if the estimator derives from ClassifierMixin..

So it's kind of automated inside the call..

Check this kaggle kernel link

$\endgroup$
6
  • $\begingroup$ Yes in here there is no pre division of data.but in mnist xtrain and xtest are separate $\endgroup$ – Boris Mar 19 '18 at 6:35
  • $\begingroup$ So we validate on a part of training set as we cant use the test set as such? $\endgroup$ – Aditya Mar 19 '18 at 6:36
  • $\begingroup$ can't we concatenate the training and testing dataset and exploit the whole dataset and use k-fold cross validation $\endgroup$ – Boris Mar 19 '18 at 6:49
  • $\begingroup$ Do you have the labels for your test set? $\endgroup$ – Aditya Mar 19 '18 at 7:39
  • $\begingroup$ yes its mnist dataset $\endgroup$ – Boris Mar 19 '18 at 9:46
0
$\begingroup$

For the MNIST data,what you need to do is , apply cross validation on your training data for checking the performance of your model. Then, If you are satisfied by the performance of the model, you can train it on the whole training set. After that, you will use the trained model to make predictions for the test dataset.

$\endgroup$
0
$\begingroup$

You can either validate your results on the test set or if you want to use KFold then you could first concatenate the train and test set first and then use KFold splitting to evaluate your results. Hope it helps!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.