3
$\begingroup$

I am new to Machine learning in that Artificial Neural Network. I am using nnet package in R for training of Neural Network. What is the difference between model parameters and hyperparameters? I have heard that the hyper parameters are set before we set the model parameters. What are the hyper parameters and model parameters of a Artificial Neural Network and when they are actually tuned? i.e. Are hyper parameters tuned in Training stage of Neural Network and model parameters in validation stage?

$\endgroup$
3
$\begingroup$

The parameters of a neural network are typically the weights of the connections. In this case, these parameters are learned during the training stage. So, the algorithm itself (and the input data) tunes these parameters.

The hyper parameters are typically the learning rate, the batch size or the number of epochs. The are so called "hyper" because they influence how your parameters will be learned. You optimize these hyper parameters as you want (depends on your possibilities): grid search, random search, by hand, using visualisations... The validation stage help you to both know if your parameters have been learned enough and know if your hyper parameters are good.

If you want to know more about hyper parameters and parameters in general in machine learning, look for "deep learning versus shallow learning".

| improve this answer | |
$\endgroup$
2
$\begingroup$

I'd characterize model parameters as the architectural choices of the neural net, i.e. how many layers, the number of nodes per layer, the type of unit (sigmoid, tanh etc.), whereas hyperparameters are things such as the learning rate, momentum, regularization coefficient and such like. The hyperparameters need to be tuned during training for any given neural net architecture.

| improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ I think the number of layers, nodes per layers and types of unit are hyper parameters too. Hyper means that they influence the way the model will be built. The parameters are just the values of your model. $\endgroup$ – Robin Mar 16 '17 at 11:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.