"I want to train a model and also perform cross-validation in scikit-learn"
By this, I assume you meant saying, "I want to assess different model performace using K-fold Cross-validation approach and based on the performance, I want to select a model fitting"
Primarily, K-fold CV will divide the training data into a number of folds k, and then will fit and evaluate the model. Reference from chapter 5 in ISLR:
Cross-validation can be used to estimate the test error associated
with a given statistical learning method in order to evaluate its
performance, or to select the appropriate level of flexibility.
I try to explain with a sample code:
from sklearn import svm
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
# Training Data
feature_cols=['X1', 'x2', 'x3']
X = train_data[feature_cols]
y = train_data.target
#Build a svm classifier model
model_svm = svm.SVC(C=100, gamma=1)
#Initialise the number of folds k for doing CV
kfold = KFold(10, False, 2)
#Evaluate the model using k-fold CV
cross_val_scores = cross_val_score(model_svm, X, y, cv=kfold, scoring='accuracy')
#Get the model performance metrics
print("Mean: " + str(cross_val_scores.mean()))
You can do k-fold CV for different model parameters say (different C and gamma values in SVM) or even different models altogether (say Logistic regression) and choose the best model.
Once you chose the best performing model you then can proceed to fit the model for making the prediction.
#Fit the model
model_svm.fit(X,y)
#Prediction on test set
model_svm.predict(test_data)
"I believe and according to my understanding if the CV is training the model for performing k-fold validation, do we need to train again as any of k-models are not trained on the whole training data."
In the above sample code, the number of folds k=10. So CV will be done 10 iterations with each training set containing (k − 1)n/k = 9(n/10) observations and n/10 observations for testing. So no, you don't need to train again as K-fold CV does this procedure for you. From ISLR:
K-fold CV approach involves randomly dividing the set of
observations into k groups, or folds, of approximately equal size. The
first fold is treated as a validation set, and the method is fit on
the remaining k − 1 folds. This procedure is
repeated k times; each time, a different group of observations is
treated as a validation set.
"My main confusion is that for hyper parameter tuning and in grid search we can have best paramters,but in cross validation only do we need to call the fit method after CV step"
Again, the k-fold CV method is used for model assessment. What you can do is to use the grid search for selecting the best parameters and say the GridSearch method to use K-fold Cross-validation to selecting the best parameters. Also in a way Yes, you will call the fit method after CV because you would have chosen the best performing method and be proceeding to fit the model.
from sklearn import svm
from sklearn.model_selection import KFold
from sklearn.model_selection import GridSearchCV
# Training Data
feature_cols=['X1', 'x2', 'x3']
X = train_data[feature_cols]
y = train_data.target
#Build a svm classifier model
model_svm = svm.SVC(C=100, gamma=1)
#Initialise the number of folds k for doing CV
kfold = KFold(10, False, 2)
Cs = [0.001, 0.01, 0.1, 0.3, 1, 3, 10, 100]
gammas = [0.001, 0.01, 0.1, 0.3, 1, 3, 10, 100]
param_grid = {'C': Cs, 'gamma' : gammas}
grid_search = GridSearchCV(model_svm, param_grid, cv=kfold, scoring='accuracy')
grid_search.fit(X, y)
grid_search.best_params_
I hope this clears thing up.
