validation after hyperparameter tuning

Question

I tuned my hyperparameter with random search and i used cv=5. Is it important to validated the hyperparameter with model and testdata or is it okay to use the given back accuracy from random search?

Answer 1

In machine learning there is a fundamental difference between what is known as hyperparameters and parameters.

Parameters

They are the variables that define your model, or in other words the relationship between your inputs and the output you are trying to predict.

For example, in simple linear regression, the goal is to be able to make predictions for a target variable $y$ given some new input value $x$ of the input variable on the basis of a set of training data comprising $N$ input values $\mathbf{x} = (x_1, \dots, x_n)^{\mathbf{T}}$. Relationship between inputs and output is given by

$$ y = w_0 x + w_1 $$

where $w_0$ and $w_1$ are the parameters our algorithm tries to "learn" during training to produce the most accurate possible predictions for $y$. FYI the term $w_0$ is called weight and the term $w_1$ is called bias.

Hyperparameters

These are different from the parameters above in the sense that they act on the training process itself. Hyperparameters can be seen as knobs you would tweak to adjust the learning of your model in the best possible way.

An example of hyperparameter is that of the learning rate $\eta$ used to adjust how far apart from each other values are computed when looking for best possible parameters of a model using Gradient Descent. Another example is the number of decision trees to have in the forest when using Random Forest algorithm.

Hyperparameter tuning

Unlike model parameters, hyperparameters can be set by the user before training a machine learning model. To be fair the Random Search Cross-Validation method you mentioned is pretty efficient for finding most suitable values for hyperparameters given a machine learning problem. You can reasonably use results from your cross validation for hyperparameters.

Next steps should include testing your model on a held-off test data to check that your model parameters determined during training are indeed correctly setting your model up.