1
$\begingroup$

Is there any pedantic difference between referring to a specific "parameter" of a model as $\theta_1$ or referring to a specific "weight" of that model as $w_1$?

Are they always the exact same thing?

$\endgroup$
2
$\begingroup$

What you are referring to as parameter is rather called hyper-parameter. I also assume that you are talking about weights as in a neural network, so that would be your model's parameters.

Basically, the difference is that hyper-parameters are chosen by the user prior to learning (and affect the learning phase), while the parameters are computed by the algorithm during the learning process. Thus, the model's parameters depend on training data and the model's hyperparameters.

Note: Things are not always that simple, because optimizing hyper-parameters is often mandatory, and in many cases you could say that they are learnt from the validation phase (model assessment over the validation dataset), so they may also depend on the dataset.

$\endgroup$
2
  • $\begingroup$ I'm curious why you think I'm referring to hyper-parameters? I am specifically looking for differentiation between parameter and weight (not hyper-parameter). As you have used the words weights and parameters (not hyper-parameters) interchangeably in your answer, I assume your final thought is that they are exactly the same thing? $\endgroup$ Nov 25 '19 at 18:50
  • $\begingroup$ In the case of neural networks, the weights are the model's parameters, indeed. In other types of models, those are simply named parameters, the word weight is restricted to a few cases, mainly neural networks. I initially thought that you were talking about hyper-parameters because of the notation: $\theta$ is the usual name given to the hyper-parameters vector. $\endgroup$ Nov 25 '19 at 18:59
0
$\begingroup$

From my experience parameters refer to high-level tuning of the algorithm, for instance, the learning rate also known as hyper-parameters. Whereas, weights are used for lower-level tuning such as weight a feature, or an instance. For example, one could increase the weight of the positive instances.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.