1
$\begingroup$

Let's assume we perform a ML based classification in terms of some supervised ML approach. In view of a bias-variance trade-off, we decide to use the k-fold cross-validation strategy. For each fold we get a confusion matrix and a classification report ...eventually a ROC-curve. Thus we get k such results. One might characterize the overall score of each such fold by the mean-squared error (MSE). Thus, we get k times different MSE scores. One might take the average of the MSE's and find the mean score of the model based on the k-fold cross-validation, MSE_mean.

Here is the code that I run in the case of SVM:

from sklearn.model_selection import KFold
from sklearn.utils import shuffle
from sklearn.svm import SVC
from sklearn.metrics import classification_report, confusion_matrix

def SVM(X_train, X_test, y_train, y_test, kernelType):
    svclassifier = SVC(kernel=kernelType)
    svclassifier.fit(X_train, y_train)
    y_pred = svclassifier.predict(X_test)
    confusionMatrix = confusion_matrix(y_test,y_pred)
    classificationReport = classification_report(y_test,y_pred)
    return confusionMatrix, classificationReport
xall, yall = shuffle(x, y, random_state=21)

kf = KFold(n_splits=5)
from imblearn.over_sampling import SMOTE
for train_index, test_index in kf.split(x):
   yall.iloc[train], yall.iloc[test]
   x_train = xall.iloc[train_index,:]
   y_train = yall.iloc[train_index]
   x_test = xall.iloc[test_index,:]
   y_test = yall.iloc[test_index]

 # smote
smote = SMOTE(sampling_strategy='minority')
x_train_sm, y_train_sm = smote.fit_sample(x_train, y_train)
x_train_sm, y_train_sm = shuffle(x_train_sm, y_train_sm, random_state=21)


#confusionMatrixSVM, classificationReportSVM 
confusionMatrixSVM, classificationReportSVM = SVM(x_train_sm, x_test, y_train_sm,   y_test, 'rbf')
print(confusionMatrixSVM)
print(classificationReportSVM)

My questions: Since we do ML in order to make predictions, we want to have a single model at the end of the k-fold cross-validation process and not separate k models. How do we get a single model at the end ? That model should in principle provide a score MSE = MSE_mean. I need that model for making predictions. Can you suggest some code for getting a single model after the k-fold cross-validation ?

Many thanks.

$\endgroup$
6
  • 2
    $\begingroup$ Does this answer your question? What is the next step after k fold CV? $\endgroup$
    – Erwan
    Commented Nov 7, 2021 at 16:55
  • $\begingroup$ This is a common confusion: k-fold CV is only used for evaluation, not for training a model. If one uses CV on the training set in order to evaluate the model, the usual method is to re-train a final model on the full training set after CV. $\endgroup$
    – Erwan
    Commented Nov 7, 2021 at 16:57
  • $\begingroup$ Thanks. You are saying on the answer to the other question,''final model as accurate as possible so we should use all the data " . But in the answer to my question, you are saying that one should re-train the model on the training set. By training set I presume you mean the set from the training/test split of the dataset.Am I wrong ? $\endgroup$
    – user127469
    Commented Nov 7, 2021 at 18:44
  • $\begingroup$ The general method is like this: 1) split between training and test set; 2) run k-fold CV on the training set (i.e. train k models, each of them trained on k-1 subsets of the training data); 3) train the final model on the full training set; 4) evaluate the final model on the test set. When I said "we should use all the data" I meant all the training data, as opposed to using one of the k models from the CV which is trained only on a subset of the training set. Note that CV is not always necessary, and also there can be cases where there's no point evaluating on a separate test set after CV. $\endgroup$
    – Erwan
    Commented Nov 7, 2021 at 20:42
  • $\begingroup$ Many thanks. If in cases there is no point evaluating the final model on a separate test set after CV, then one will train the model on the whole dataset, right? What are the criteria for avoiding the evaluation on a test set after CV, i.e. can you name a scenario where this is possible ? $\endgroup$
    – user127469
    Commented Nov 8, 2021 at 3:49

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.