Could I directly apply techniques for hyper-parameter tuning, and choose the best model?

Question

I have noticed in some sources the author first trains the model (say a model from scikit-learn) with the default hyper-parameters, and the model naturally gives a result. Then, they would try to optimize the hyper-parameters, even if the parameter grid is containing the same default parameters (for example with Exhaustive Grid Search), and then the optimal model is chosen with the best parameters.

While I was practicing, I have done the same steps, but after I dissected the process, I realized that this is probably redundant. If Exhaustive Grid Search (or any other technique) involves training the model with various combinations of hyper-parameters, isn't it more reasonable to directly use these techniques, and directly obtain the best model for the problem, instead of trying to tune the model with the default hyper-parameters, which will almost always results in an improved performance. Like this piece of code, from the official site of scikit-learn. Fit the pre-processed training dataset with the best model, and move on from here with the project:

parameters = {'kernel':('linear', 'rbf'), 'C':[1, 10]}
svc = svm.SVC()
clf = GridSearchCV(svc, parameters)
clf.fit(iris.data, iris.target)

Additionally, are there cases where tuning the models would not be the wisest idea, or no matter what, always tuning the models is the best practice?

Answer 1

Building model is an iterative process, we cannot guess which algorithm and which parameters give good results to your data in the beginning itself. So it is always good to start with the model with default parameters and then go for tuning the parameters.

Also, the default parameters are not same as the one you are providing in GridSearchCV. For example, In SVC, the default kernel is 'rbf' and it need gamma parameter as well. Still you can get good results using 'linear' kernel which will be much faster compared to other kernels.

It all depends on the data you are using. I could see from the question that you are using iris data which have only less features; but, if going to build a model of 1,000 features using SVC then default parameters will not help you there. Then you have to try various combinations of parameters using GridSearchCV.

Finally, I conclude that always tuning the model is the best practice. You cannot skip this.