Cross validation and hyperparameter tuning workflow

Question

After reading a lot of articles on cross validation, I am now confused. I know that cross validation is used to get an estimate of model performance and is used to select the best algorithm out of multiple ones. After selecting the best model (by checking the mean and standard deviation of CV scores) we train that model on the whole of the dataset (train and validation set) and use it for real world predictions.

Let's say out of the 3 algorithms I used in cross validation, I select the best one. What I don't get is in this process, when do we tune the hyperparameters? Do we use Nested Cross validation to tune the hyperparameters during the cross validation process or do we first select the best performing algorithm via cross validation and then tune the hyperparameter for only that algorithm?

PS: I am splitting my dataset into train, test and valid where I use train and test sets for building and testing my model (this includes all the preprocessing steps and nested cv) and use the valid set to test my final model.

Edit 1 Below are two ways to perform Nested cross validation. Which one is the correct way aka which method does not lead to data leakage/overfitting/bias?

Method 1: Perform Nested CV for multiple algorithms and their hyperparameters simultaneously:-

from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import mean_squared_error
from sklearn.ensemble import RandomForestRegressor
from sklearn.svm import SVR
from sklearn.datasets import make_regression
import numpy as np
import pandas as pd

# create some regression data
X, y = make_regression(n_samples=1000, n_features=10)

# setup models, variables
results = pd.DataFrame(columns = ['model', 'params', 'mean_mse', 'std_mse'])
models = [SVR(), RandomForestRegressor(random_state = 69)]
params = [{'C':[0.01,0.05]},{'n_estimators':[10,100]}]

# split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.3)

# estimate performance of hyperparameter tuning and model algorithm pipeline
for idx, model in enumerate(models):
    
    # perform hyperparameter tuning
    clf = GridSearchCV(model, params[idx], cv = 3, scoring='neg_mean_squared_error')
    clf.fit(X_train, y_train)

    # this performs a nested CV in SKLearn
    score = cross_val_score(clf, X_train, y_train, cv = 3, scoring='neg_mean_squared_error')
    
    row = {'model' : model,
           'params' : clf.best_params_,
           'mean_mse' : score.mean(),
           'std_mse' : score.std()}

    # append the results in the empty dataframe
    results = results.append(row, ignore_index = True)

Method 2: Perform Nested CV for single algorithm and it's hyperparameters:-

from sklearn.datasets import load_iris
from matplotlib import pyplot as plt
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV, cross_val_score, KFold, train_test_split
import numpy as np

# Load the dataset
iris = load_iris()
X_iris = iris.data
y_iris = iris.target

train_x, test_x, train_y ,test_y = train_test_split(X_iris, y_iris, test_size = 0.2, random_state = 69)

# Set up possible values of parameters to optimize over
p_grid = {"C": [1, 10], "gamma": [0.01, 0.1]}

# We will use a Support Vector Classifier with "rbf" kernel
svm = SVC(kernel="rbf")

# Choose cross-validation techniques for the inner and outer loops,
# independently of the dataset.
# E.g "GroupKFold", "LeaveOneOut", "LeaveOneGroupOut", etc.
inner_cv = KFold(n_splits=4, shuffle=True, random_state=69)
outer_cv = KFold(n_splits=4, shuffle=True, random_state=69)
    
# Nested CV with parameter optimization
clf = GridSearchCV(estimator=svm, param_grid=p_grid, cv=inner_cv)
clf.fit(train_x, train_y)
nested_score = cross_val_score(clf, X=X_iris, y=y_iris, cv=outer_cv)
        
nested_scores_mean = nested_score.mean()
nested_scores_std = nested_score.std()

Answer 1

Suppose you have two models which you can choose $m_1$, $m_2$. For a given problem, there is a best set of hyperparameters for each of the two models (where they perform as good as possible), say $m_1^*$, $m_2^*$. Now say $Acc(m_1^*) > Acc(m_2^*)$, i.e. model 1 is better than model 2.

Now suppose you have tuned model 2 (or you have "okay" hayperparameter by coincidence) but you use inferior hyperparameter for model 1. You could end up finding $Acc(m_1^s) < Acc(m_2^*)$ (i.e. "choose model 2"), while the true best choice would be: "use tuned model 1".

Thus in order to make an informed decision, you would need to "tune" both models and compare the performance of the tuned models with "best hyperparameter". What I often do is to define test and train data, tune possible models using cross validation (train data only!), and assess the performance of the tuned models based on the test set.

In addition you may want to do feature engineering / feature generation. This should be done before tuning the models, since different data may lead to different optimal hyperparameter, e.g. in case of a random forest, where the number of split candidates per split can be contingent on the number and the quality of features.