# Which parameters are hyper parameters in a linear regression?

Can the number of features used in a linear regression be regarded as a hyperparameter? Perhaps the choice of features?

• It's for feature selection phase. Welcome to our community :) May 14, 2018 at 1:53

I like the way Wikipedia generally defines it:

In machine learning, a hyperparameter is a parameter whose value is set before the learning process begins. By contrast, the values of other parameters are derived via training.

On top of what Wikipedia says I would add:

Hyperparameter is a parameter that concerns the numerical optimization problem at hand. The hyperparameter won't appear in the machine learning model you build at the end. Simply put it is to control the process of defining your model. For example like in many machine learning algorithms we have learning rate in gradient descent (that need to be set before the learning process begins as Wikipedia defines it), that is a value that concerns how fast we want the gradient descent to take the next step during the optimization.

Similarly as in Linear Regression, hyperparameter is for instance the learning rate. If it is a regularized Regression like LASSO or Ridge, the regularization term is the hyperparameter as well.

Number of features: I would not regard "Number of features" as hyperparameter. You may ask yourself whether it is a parameter you can simply define during the model optimization? How you set the Number of features beforehand? To me "Number of features" is part of feature selection i.e. feature engineering that goes before you run your optimization! Think of image preprocessing before building a deep neural network. Whatever image preprocessing is done is never considered hyperparameter, it is rather a feature engineering step before feeding it to your model.

• Nice answer. Wonder if in Linear Regression, the order of the polynomial is a hyperparameter? Jul 18, 2020 at 16:32
• Thanks. This is similar to the comment I gave about "Number of features"! I wouldn't count the "order of the polynomial" as a hyperparameter! Again Wikipedia adds a good explanation: "For example - if we treat the degree of a polynomial equation fitting a regression model as a trainable parameter - this would just raise the degree up until the model perfectly fit the data, giving small training error - but bad generalization performance." (en.wikipedia.org/wiki/Hyperparameter_(machine_learning)). Hope this helps. Jul 19, 2020 at 9:50
• Wait, isn’t a hyperparameter something that is NOT learnt? I mean I get the first part of your comment, not the second one. But even for the first part - if regularisation parameter is an hyperparameter, then similarly the order of the polynomial could be seen as such? Jul 19, 2020 at 11:45
• Good answer, except that I don't understand "as in Linear Regression, hyperparameter is for instance the learning rate". What is "learning rate" in linear regression? Mar 10, 2023 at 0:47
• It is a Gradient Descent optimization, and there is a learning rate what decides how fast you want to update the weights, see analyticsvidhya.com/blog/2021/04/…, you Google, you find lots of articles describing it eveb better, but it is standard Mar 10, 2023 at 7:55

Hyper-parameters by definition are input parameters which are necessarily required by an algorithm to learn from data.

For standard linear regression i.e OLS, there is none. The number/ choice of features is not a hyperparameter, but can be viewed as a post processing or iterative tuning process.

On the other hand, Lasso takes care of number/choice of features in its formulation of the loss function itself, so only hyper-parameter for it would be the shrinkage factor i.e lambda

• Hyperparameters are not learning from data. your first statement's last few words need to be changed. Nov 16, 2022 at 17:11

The features from your data set in linear regression are called parameters. Hyperparameters are not from your data set. They are tuned from the model itself. For example, the level of splits in classification models.

For basic straight line linear regression, there are no hyperparameter.