I am new to Machine learning in that Artificial Neural Network. I am using nnet package in R for training of Neural Network. What is the difference between model parameters and hyperparameters? I have heard that the hyper parameters are set before we set the model parameters. What are the hyper parameters and model parameters of a Artificial Neural Network and when they are actually tuned? i.e. Are hyper parameters tuned in Training stage of Neural Network and model parameters in validation stage?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
The parameters of a neural network are typically the weights of the connections. In this case, these parameters are learned during the training stage. So, the algorithm itself (and the input data) tunes these parameters.
The hyper parameters are typically the learning rate, the batch size or the number of epochs. The are so called "hyper" because they influence how your parameters will be learned. You optimize these hyper parameters as you want (depends on your possibilities): grid search, random search, by hand, using visualisations... The validation stage help you to both know if your parameters have been learned enough and know if your hyper parameters are good.
If you want to know more about hyper parameters and parameters in general in machine learning, look for "deep learning versus shallow learning".
-
$\begingroup$ Why call them "parameters" instead of "weights", as the have been traditionally called? The switches in terminology sometimes drives me nuts! :) $\endgroup$ Commented Nov 26, 2021 at 20:25
-
$\begingroup$ I'm not sure they've always been called "weights" instead of "parameters" in all fields. Since it is a rather new area of research/engineering, I guess it normal to have these little differences in terminology since they evolve really fast these days ! From a pure machine learning perspective, talking of "parameters" make more sens than "weights" when comparing with SVMs or other models. $\endgroup$– RobinCommented Jan 18, 2022 at 15:34
-
$\begingroup$ New? Neural nets have been around since Perpetron and in the 80s, I was around when backpropagation was being developed. Not new at all. $\endgroup$ Commented Mar 21 at 0:13
$\begingroup$
$\endgroup$
1
I'd characterize model parameters as the architectural choices of the neural net, i.e. how many layers, the number of nodes per layer, the type of unit (sigmoid, tanh etc.), whereas hyperparameters are things such as the learning rate, momentum, regularization coefficient and such like. The hyperparameters need to be tuned during training for any given neural net architecture.
-
2$\begingroup$ I think the number of layers, nodes per layers and types of unit are hyper parameters too. Hyper means that they influence the way the model will be built. The parameters are just the values of your model. $\endgroup$– RobinCommented Mar 16, 2017 at 11:14