1
$\begingroup$

I am looking into hyper-parameter tunning and was curious about whether the search space is considered continuous or discrete? My understanding of both those cases: 1. Continuous would make it 'easier' to look for hyperparameters combinations that are more fine-tuned to the problem. 2. Discrete would imply that a more brute-force (or random) approach would be helpful.

Subsidiary question: Am I mistaking on my understanding of either of those cases?

Thanks

$\endgroup$
1
$\begingroup$

Continuous means only you have continuous variables. It can be convex or concave. It might not even be differentiable. Gradient descent only applies to differentiable convex problems (or convex approximations of concave problems).

Example:

  • learning rate of SGD is continuous hyper parameter.
  • NN optimizer (SGD, Adam, RMSprop) is discrete parameter

From my perspective: right now, rarely we have resources to perform cross validation on a number of deep models. Most of the hyper parameters are tuned using intuition, a bit of math and tuned in isolation.

If you can afford it:

  • grid search to initialize solutions
  • if your'e unhappy with the results:
    • run some hill climber on space, like simulated annealing
    • run simple GA, there'll be problems because of mixed space (some params are continuous, some are discrete), you'll need custom operators
$\endgroup$
3
  • $\begingroup$ This raised me a question. Let's put per example, finding the right penalty for a lasso regression. The loss function is differentiable (it is the one of the lasso) and the different parameters are continous. You could perform there a gradiente descent to see which one is the best parameter. If it was discrete you couldn't perform a step (only if it was a fixed step). Right? $\endgroup$ – Carlos Mougan Mar 18 '20 at 10:54
  • $\begingroup$ When refering to NN optimizer to what do you refer? The learning rate? or the algorithm $\endgroup$ – Carlos Mougan Mar 18 '20 at 11:05
  • 1
    $\begingroup$ An algorithm. The particular algorithm you choose is a discrete hyper-param. $\endgroup$ – Piotr Rarus Mar 18 '20 at 11:28
-1
$\begingroup$

Yep, you are understanding it right.

The Hyperparameter search is another optimization problem. You have to search along with all the parameters.

If it was continuous you could apply gradient descent for example. But since it is discrete you can´t search the space this way. There are other strategies considered here, random search, grid search, Bayesian or even genetic algorithms to try to find the optimal parameters

$\endgroup$
1
  • $\begingroup$ sry, but continuous works different ;) $\endgroup$ – Piotr Rarus Mar 18 '20 at 10:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.