# Is the search space of Hyperparameters Continuous or Discrete?

I am looking into hyper-parameter tunning and was curious about whether the search space is considered continuous or discrete? My understanding of both those cases: 1. Continuous would make it 'easier' to look for hyperparameters combinations that are more fine-tuned to the problem. 2. Discrete would imply that a more brute-force (or random) approach would be helpful.

Subsidiary question: Am I mistaking on my understanding of either of those cases?

Thanks

Continuous means only you have continuous variables. It can be convex or concave. It might not even be differentiable. Gradient descent only applies to differentiable convex problems (or convex approximations of concave problems).

Example:

• learning rate of SGD is continuous hyper parameter.
• NN optimizer (SGD, Adam, RMSprop) is discrete parameter

From my perspective: right now, rarely we have resources to perform cross validation on a number of deep models. Most of the hyper parameters are tuned using intuition, a bit of math and tuned in isolation.

If you can afford it:

• grid search to initialize solutions
• if your'e unhappy with the results:
• run some hill climber on space, like simulated annealing
• run simple GA, there'll be problems because of mixed space (some params are continuous, some are discrete), you'll need custom operators
• This raised me a question. Let's put per example, finding the right penalty for a lasso regression. The loss function is differentiable (it is the one of the lasso) and the different parameters are continous. You could perform there a gradiente descent to see which one is the best parameter. If it was discrete you couldn't perform a step (only if it was a fixed step). Right? Mar 18, 2020 at 10:54
• When refering to NN optimizer to what do you refer? The learning rate? or the algorithm Mar 18, 2020 at 11:05
• An algorithm. The particular algorithm you choose is a discrete hyper-param. Mar 18, 2020 at 11:28

Yep, you are understanding it right.

The Hyperparameter search is another optimization problem. You have to search along with all the parameters.

If it was continuous you could apply gradient descent for example. But since it is discrete you can´t search the space this way. There are other strategies considered here, random search, grid search, Bayesian or even genetic algorithms to try to find the optimal parameters

• sry, but continuous works different ;) Mar 18, 2020 at 10:17