Which is first ? Tuning the parameters or selecting the model

Question

I've been reading about how we split our data into 3 parts; generally, we use the validation set to help us tune the parameters and the test set to have an unbiased estimate on how well does our model perform and thus we can compare models based on the result of the test set. However, i've also read that model selection shoud be done before tuning the parameters. I'm getting confused. Which one must be done before the other ? Is the validation set used for tuning ? If true, how are we supposed to do model selection before tuning the parameters ?

Answer 1

You can tune parameters only if you have already trained the model, otherwise there is nothing to tune.

However, i've also read that model selection shoud be done before tuning the parameters.

Before tuning you need to do some kind of pre-processing before tuning the parameters. Usually your pipeline will consist of:

1. Get Data and Clean It.
2. Do some EDA ( Exploritary Data Analysis) do get better understanding of you data.
3. Do some feature engineering - roughly, if possible, you would like to transform current data, so it would be more suitable for your ML algo.
4. Train and evaluate your model on validation data.
5. Tune your model`s params to get better performance ( fight with overfitting alos included).
6. Evaluate on Test set

how are we supposed to do model selection before tuning the parameters

About Model Selection - I think even without having deep inshights about your data, you can already choose some baseline models (I think its a good rule to start with simple models like Linear Regression or K-NN to get the "feel of the performance"), because in most cases you know that you are probably dealing with regression/classification/clustering task, so you can already specify the set on models to try - but before trying the horde of ML models, you should get your data ready.

Some posts like this one can give you the feels of available ml algos, but there are a lot more to try.

In Addition, I think this paper might be very usefull if you are starting to work with ML

Answer 2

However, i've also read that model selection shoud be done before tuning the parameters. [...] If true, how are we supposed to do model selection before tuning the parameters ?

Of course model selection should be done before tuning the parameter.
Imagine you chose a model, then work on tuning the parameter (it requires a lot of time, of course). The results are good. But then you test another model, and even without tuning the parameters, the results are better. All your work done with the first model is useless, you spending days, maybe weeks to tune a model that do not perform best on your data.

Model selection come first, because it's useless to tune parameters for a model that you won't keep. Of course, maybe you don't know which model to chose before testing it with your data. You have to guess which one perform best, or maybe try a few and see which is better, even with a few dataset or a few training epoch.

Is the validation set used for tuning ?

The validation set can be used to tune the model's hyperparameters. Generally, you test a subset of hyperparameters, train each model on a few epoch, and see which perform best. Then keep this set of hyperparameters, and train the model with the real dataset, as much as require to have good performances.

Of course, to have the best model, you need to test them all, with all hyperparameters possible. But you can't do that, so you need to optimise your training, by selecting a model first, then tune it with a few subsest of hyperparameters

Answer 3

The process of selecting model is as follows:

1] Choose the appropriate algorithm to suite your data set.

2] Create the model, in this process we will fit the algorithm with training data along with the few other machine learning techniques like grid search and cross validation.If you are using deep learning then you might need to split the data into training, validation and testing data. In machine learning if you are using cross validation you do not need the validation data as cross validation it self will split data for validation.

3] Now, depending on the outcome of the result of the above step, you tune hyper parameters and check again the results again. You can repeat the process until you get the optimal model with best metrics. In this way you are selecting the best model by hyper parameter tuning.

One more thing about selecting the model, you can change the algorithm in the first step and continue with step2 and step3. Now, you will have different models with their own best optimal model with best hyperparameters. You can then select best model that suits your needs.

For Example

1] First you select the decision tree to create the model and then select the best optimal model with hyper parameter tuning using step2 and step3.

2] Second, you change the algrithm to randomforest to create the model and then again select the best optimal model with hyper parameter tuning using step2 and step3.

3] Now we will have different models, each of this model was selected as the best model using the hyper parameter tuning. In this way you can try creating the few models and compare them and then finally select the best model.

So, there are actually 2 scenarios when selecting the model.

1] You create the model with an algorithm and then select the best optimal model with hyperparameter tuning.

2] You create different models with different algorithms each with their own best optimal model and select the best model out of these.

However, i've also read that model selection shoud be done before tuning the parameters. I'm getting confused. Which one must be done before the other ?

Sometimes, we might get confused with creating the model and selecting the model. When we create the model, we have to select the appropriate algorithm, sometimes this step(where we select the algorithm) can also be referred to as selecting the model. So, hyper parameter helps in selecting the best model, therefore hyper parameter tuning comes before selecting the model.

Is the validation set used for tuning ?

Validation set is used to evaluate the model performance before testing on the testing data. Once we evaluate the model performance and if it is not as good as you expected, then you change the hyper parameters and check the performance again. So, yes we will use the validation data set to tune hyper parameters.

You can think of validation data as doing unit testing while developing the software applications. You develop the application as a developer and do the unit testing. If the unit testing fails, then again you have to make changes as the developer to fix the unit testing bugs and then only it will be tested by testing team. Once the testing team approves, then the application will be deployed in the production. Similarly, in machine learning you develop the model with training data and do the first testing on validation data. Then you do hyper parameter tuning and select the best model. This model will be tested on the testing data set. Once the results on the testing data set are satisfying, you will deploy the model in the production.

Answer 4

Model parameters are estimated from data automatically and model hyperparameters are set manually and are used in processes to help estimate model parameters.

Model hyperparameters are often referred to as parameters because they are the parts of the machine learning that must be set manually and tuned.

Basically, parameters are the ones that the “model” uses to make predictions etc. For example, the weight coefficients in a linear regression model. Hyperparameters are the ones that help with the learning process. For example, number of clusters in K-Means, shrinkage factor in Ridge Regression. They won’t appear in the final prediction piece, but they have a large influence on how the parameters would look like after the learning step.

Refer : https://machinelearningmastery.com/difference-between-a-parameter-and-a-hyperparameter/