0
$\begingroup$

I want to train a model and also perform cross validation in scikit-learn, If i want to access the model (For instance to see the parameter's selected and weights or to predict) i will need to fit it again.

I believe and according to my understanding if CV is training the model for performing k-fold validation, do we need to train again as any of k-models are not trained on the whole training data.

I have seen cross_val_predict method and it is not exactly what i need as it is for predicting score and i want model object.

My main confusion is that for hyper parameter tuning and in grid search we can have best paramters,but in cross validation only do we need to call the fit method after CV step

$\endgroup$

1 Answer 1

1
$\begingroup$

"I want to train a model and also perform cross-validation in scikit-learn"

By this, I assume you meant saying, "I want to assess different model performace using K-fold Cross-validation approach and based on the performance, I want to select a model fitting"

Primarily, K-fold CV will divide the training data into a number of folds k, and then will fit and evaluate the model. Reference from chapter 5 in ISLR:

Cross-validation can be used to estimate the test error associated with a given statistical learning method in order to evaluate its performance, or to select the appropriate level of flexibility.

I try to explain with a sample code:

from sklearn import svm
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score

# Training Data
feature_cols=['X1', 'x2', 'x3']
X = train_data[feature_cols]
y = train_data.target

#Build a svm classifier model
model_svm = svm.SVC(C=100, gamma=1)

#Initialise the number of folds k for doing CV
kfold = KFold(10, False, 2)

#Evaluate the model using k-fold CV
cross_val_scores = cross_val_score(model_svm, X, y, cv=kfold, scoring='accuracy')

#Get the model performance metrics
print("Mean: " + str(cross_val_scores.mean()))

You can do k-fold CV for different model parameters say (different C and gamma values in SVM) or even different models altogether (say Logistic regression) and choose the best model.

Once you chose the best performing model you then can proceed to fit the model for making the prediction.

#Fit the model
model_svm.fit(X,y)

#Prediction on test set
model_svm.predict(test_data)

"I believe and according to my understanding if the CV is training the model for performing k-fold validation, do we need to train again as any of k-models are not trained on the whole training data."

In the above sample code, the number of folds k=10. So CV will be done 10 iterations with each training set containing (k − 1)n/k = 9(n/10) observations and n/10 observations for testing. So no, you don't need to train again as K-fold CV does this procedure for you. From ISLR:

K-fold CV approach involves randomly dividing the set of observations into k groups, or folds, of approximately equal size. The first fold is treated as a validation set, and the method is fit on the remaining k − 1 folds. This procedure is repeated k times; each time, a different group of observations is treated as a validation set.

"My main confusion is that for hyper parameter tuning and in grid search we can have best paramters,but in cross validation only do we need to call the fit method after CV step"

Again, the k-fold CV method is used for model assessment. What you can do is to use the grid search for selecting the best parameters and say the GridSearch method to use K-fold Cross-validation to selecting the best parameters. Also in a way Yes, you will call the fit method after CV because you would have chosen the best performing method and be proceeding to fit the model.

    from sklearn import svm
    from sklearn.model_selection import KFold
    from sklearn.model_selection import GridSearchCV

    # Training Data
    feature_cols=['X1', 'x2', 'x3']
    X = train_data[feature_cols]
    y = train_data.target

    #Build a svm classifier model
    model_svm = svm.SVC(C=100, gamma=1)

    #Initialise the number of folds k for doing CV
    kfold = KFold(10, False, 2)

    Cs = [0.001, 0.01, 0.1, 0.3, 1, 3, 10, 100]
    gammas = [0.001, 0.01, 0.1, 0.3, 1, 3, 10, 100]

    param_grid = {'C': Cs, 'gamma' : gammas}
    grid_search = GridSearchCV(model_svm, param_grid, cv=kfold, scoring='accuracy')
    grid_search.fit(X, y)
    grid_search.best_params_

I hope this clears thing up.

$\endgroup$
2
  • $\begingroup$ I appreciate your answer. Still we can select a model from hyperparamter tuning, but in case of CV we dont get a fitted model and only get scores? $\endgroup$
    – A.B
    Aug 2, 2019 at 10:36
  • $\begingroup$ Yes, I agree. With CV you only get scores. The idea is to use these CV scores for estimating bias-variance tradeoff. But with gridsearch, you be getting a fitted model with the best hyperparameters. gridsearch = GridSearchCV(model_svm, hyperparameters, cv=10, verbose=1) best_model = gridsearch.fit(X, target) $\endgroup$
    – satznova
    Aug 2, 2019 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.