# Grid search or gradient descent?

Assume we have a neural network and one if its activation functions is a function of parameter a. We want to find the weights and parameter a that leads to the minimum loss on the validation set which one is better?:

1) Treat a as a hyperparameter. Do grid search for a: consider range for a and evaluate the loss for a few points in the range. Pick the best a in the range along with the weights leading to the minimum loss.

2) Treat a as a parameter. Since loss of the network is a function of parameter a, use gradient descent to update weights as well as parameter a at each iteration.

To me, the second option is better since it can lead to the optimal point but the grid cannot lead to the optimal point, it can lead to the points around the optimal point depending on your selection. But, is the second method right?

Is there a better method?

Lastly, in grid search, how do we pick the range?

The main distinction to be made here is between a parameter and a hyperparameter; once we have this clarified, the rest is easy: grid search is not used for tuning the parameters (only the hyperparameters), which are tuned with with gradient descend.

Now, roughly speaking, a parameter is something that changes during training; in a neural network, the only parameters are the weights and the biases, and they are tuned with gradient descend.

A hyperparameter can be thought as something "structural", e.g. the number of layers, the number of nodes for each layer (notice that these two determine indirectly also the number of parameters, i.e. how many weights and biases there are in our model), i.e. things that do not change during training. Hyperparameters are not confined to the model itself, they are also applicable to the learning algorithm used (e.g. optimization algorithm, learning rate etc). A specific set of hyperparameters defines a family of models, which differ among themselves in the exact values of their parameters; in contrast, a specific parameter set (e.g. weights & biases in a NN) define a unique model.

Having clarified the above, it should be easy to see that a in your example above is a hyperparameter, and not a parameter; as such, it would normally be optimized using grid search.

To me, the second option is better since it can lead to the optimal point

Not so fast; gradient descend is not applicable everywhere: there are mathematical prerequisites for a function in order to be eligible for optimization using gradient descend, i.e. to be continuous & defferentiable. This is indeed the case for the loss as a function of the weights & biases, but not clear if it is also the case for the loss as a function of your parameter a. And if it isn't, we simply cannot use gradient descend for tuning a.

On the contrary, techniques like grid search do not have such prerequisites, thus they can be used for a much wider range of cases.

in grid search, how do we pick the range?

As we pick almost everything else in ML & DL: empirically, and with trial & error (there is actually not much theory behind it).

UPDATE (after comment):

Now assume that the prerequisites are met for a so that we can use gradient descent. Which one is better now? Grid search or gradient descent?

Sorry, I didn't realize that this was a "Superman vs. Batman" question... Well, gradient descend is a disciplined mathematical operation, which is guaranteed to find the global minimum for convex functions (although NN loss functions are not convex), while grid search is actually just an ad hoc, quick-and-dirty procedure which does not guarantee anything (as you have already correctly suspected), so...

Beware though, especially if this is part of any homework or exam: the question, despite its phrasing, is not actually asking you which one is "better" in an abstract & hypothetical context, but which one is better in a very specific context, i.e a neural network model; and if something is not even applicable, it cannot be actually better, right?

In other words, if someone is actually testing your ML & NN knowledge here, they are certainly more interested in the exposition above, rather that which one is "better" in general; and choice (2) would be a huge mistake (after all, assume that the NN loss is a continuous & differentiable function of a is not part of the question)...

• Thanks for your answers. It was helpful. Now assume that the prerequisites are met for a so that we can use gradient descent. Which one is better now? Grid search or gradient descent? – Albert Oct 28 '19 at 22:13
• Well, you can make "a" part of the model so that it would not be a hyperparameter anymore. – Albert Oct 28 '19 at 22:13
• @Albert re your 2nd comment, I have clearly said in the answer that making a part of the model is not a trick or something; if the loss as a function of a is not continuous & differentiable, gradient descent is not applicable. – desertnaut Oct 28 '19 at 22:17