1
$\begingroup$

I am reading "A Course in Machine Learning" and, in chapter 2, the author says:

"For most models, there will be associated parameters. These are the things that we use the data to decide on. Parameters in a decision tree include: the specific questions we asked, the order in which we asked them, and the classification decisions at the leaves."

My question is about the first sentence. Is there any model in machine learning that does not have parameters? I can't think of any. For sure there are models without hyperparameters (for instance, the linear model does not contain any hyperparameter, but it still contains 2 parameters, the slope and the y-intercept).

If such parameterless models exist, what are their purpose then? Isn't it the whole point of training to tune a model's parameters?

$\endgroup$
7
$\begingroup$

Is there any model in machine learning that does not have parameters?

Yes. k-nearest neighbors is parameterless (there is only a single hyper-parameter $k$).

If such parameterless models exist, what are their purpose then? Isn't it the whole point of training to tune a model's parameters?

Exactly: such models require no training at all. k-NN in particular relies on knowing the data set upon prediction. Anything close to "training" this model would be pushing data points to a set, but these do not count as parameters.

|improve this answer|||||
$\endgroup$
1
$\begingroup$

Consider the case of the majority rule. In the majority rule you go over the training set, check which concept value is the majority and returns it for every sample.

There are no parameters, there is a training process and the classifier is has some value (mostly as a benchmark or in extreme cases).

Note the the word "parameter" has different meanings and it is hard to know what one means without context.

One might refer to hyper-parameters, like k in k-nn.

One can also refer to "problem definition" parameters, like the distance function used in k-nn.

A different meaning is actually the content of the model. I'm not sure which of the two meaning is the one the quote is using and whether the the order in which we asked them is the feature selection method (meaning 2) or the path to the leaf (meaning 3)

The last (and very common definition) is of a parameter in a Parametric model which is "is a family of distributions that can be described using a finite number of parameters. These parameters are usually collected together to form a single k-dimensional parameter vector θ = (θ1, θ2, …, θk)." Obviously, this is not the meaning used in the quote.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.