2
$\begingroup$

How to use k-fold cross validation for MNIST dataset? I read article documentation on sci-kit learn ,in that example they used the whole iris dataset for cross validation.

from sklearn.model_selection import cross_val_score
clf = svm.SVC(kernel='linear', C=1)
scores = cross_val_score(clf, iris.data, iris.target, cv=5)
scores                                              

for example while importing mnist dataset in keras

from keras.datasets import mnist
(Xtrain,Ytrain),(Xtest,Ytest)=mnist_load()

in this dataset is already divided in test and train , so to apply cross validation on the entire dataset do we need to make Xtrain and Xtest as one entity to exploit the whole data.

$\endgroup$
6
  • $\begingroup$ Split your training set using sklearn train_test_split and making a validation set out of the training set itself or go for stratified sampling X_train, X_val, Y_train, Y_val = train_test_split(X_train, y_train, test_size = 0.1, random_state=42) $\endgroup$
    – Aditya
    Commented Mar 19, 2018 at 6:31
  • $\begingroup$ I don't want to split the data it will be taken by the crossvalidation function. $\endgroup$
    – Boris
    Commented Mar 19, 2018 at 6:36
  • $\begingroup$ Yes it will be automatically done for you inside $\endgroup$
    – Aditya
    Commented Mar 19, 2018 at 6:38
  • $\begingroup$ So there is no need of train_test_split $\endgroup$
    – Boris
    Commented Mar 19, 2018 at 6:39
  • 1
    $\begingroup$ You should not use the test set for cross validation. Use a part of the training set only. This ensures that you are not overfitting to the test data. $\endgroup$
    – Jon Nordby
    Commented Apr 14, 2019 at 19:24

3 Answers 3

1
$\begingroup$

For the MNIST data,what you need to do is , apply cross validation on your training data for checking the performance of your model. Then, If you are satisfied by the performance of the model, you can train it on the whole training set. After that, you will use the trained model to make predictions for the test dataset.

$\endgroup$
0
$\begingroup$
from sklearn.model_selection import cross_val_score
clf = svm.SVC(kernel='linear', C=1)
scores = cross_val_score(clf, iris.data, iris.target, cv=5)
scores

They are not using the whole data for cross validation as such ( it's just an illusion)

When the cv argument is an integer, cross_val_score uses the KFold or StratifiedKFold strategies by default, the latter being used if the estimator derives from ClassifierMixin..

So it's kind of automated inside the call..

Check this kaggle kernel link

$\endgroup$
6
  • $\begingroup$ Yes in here there is no pre division of data.but in mnist xtrain and xtest are separate $\endgroup$
    – Boris
    Commented Mar 19, 2018 at 6:35
  • $\begingroup$ So we validate on a part of training set as we cant use the test set as such? $\endgroup$
    – Aditya
    Commented Mar 19, 2018 at 6:36
  • $\begingroup$ can't we concatenate the training and testing dataset and exploit the whole dataset and use k-fold cross validation $\endgroup$
    – Boris
    Commented Mar 19, 2018 at 6:49
  • $\begingroup$ Do you have the labels for your test set? $\endgroup$
    – Aditya
    Commented Mar 19, 2018 at 7:39
  • $\begingroup$ yes its mnist dataset $\endgroup$
    – Boris
    Commented Mar 19, 2018 at 9:46
0
$\begingroup$

You can either validate your results on the test set or if you want to use KFold then you could first concatenate the train and test set first and then use KFold splitting to evaluate your results. Hope it helps!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.