36
votes
Accepted
What is the difference between model hyperparameters and model parameters?
Hyperparameters and parameters are often used interchangeably but there is a difference between them. You can call something a 'hyperparameter' if it cannot be learned within the estimator directly. ...
- 3,021
33
votes
What is the difference between model hyperparameters and model parameters?
In addition to the answer above.
Model parameters are the properties of the training data that are learnt during training by the classifier or other ml model. For example in case of some NLP task: ...
- 2,147
27
votes
How to set the number of neurons and layers in neural networks
The consideration of the number of neurons for each layer and number of layers in fully connected networks depends on the feature space of the problem. For illustrating what happens in the two ...
- 13.9k
17
votes
What is the difference between model hyperparameters and model parameters?
Hyper-parameters are those which we supply to the model, for example: number of hidden Nodes and Layers,input features, Learning Rate, Activation Function etc in Neural Network, while Parameters are ...
- 171
17
votes
How to set the number of neurons and layers in neural networks
Very good question, as there doesn't exist an exact answer to this question yet. This is an active field of research.
Ultimately, the architecture of your network is related to the dimensionality of ...
- 5,456
12
votes
How to set the number of neurons and layers in neural networks
Short Answer: It is very related to the dimensions of your data and the type of the application.
Choosing the right number of layers can only be achievable with practice. There is no general answer ...
- 1,225
10
votes
Choosing a learning rate
Copy-pasted from my masters thesis:
If the loss does not decrease for several epochs, the learning rate might be too low.
The optimization process might also be stuck in a local minimum.
Loss being ...
- 18.8k
9
votes
How can you decide the window size on a pooling layer?
When you get a bit more insight into network topologies these hyperparameters will make more sense, but in general this is just like any other hyperparameter, you will have to test some settings and ...
- 9,348
9
votes
Accepted
Which parameters are hyper parameters in a linear regression?
I like the way Wikipedia generally defines it:
In machine learning, a hyperparameter is a parameter whose value is
set before the learning process begins. By contrast, the values of
other ...
- 4,207
9
votes
Accepted
Which is first ? Tuning the parameters or selecting the model
You can tune parameters only if you have already trained the model, otherwise there is nothing to tune.
However, i've also read that model selection shoud be done before tuning the parameters.
...
- 558
8
votes
What is the difference between model hyperparameters and model parameters?
In machine learning, a model $M$ with parameters and hyper-parameters looks like,
$Y \approx M_{\mathcal{H}}(\Phi | D)$
where $\Phi$ are parameters and $\mathcal{H}$ are hyper-parameters. $D$ is ...
- 1,313
8
votes
Accepted
How to implement Python's MLPClassifier with gridsearchCV?
A tuple of the form $(i_1, i_2, i_3, ... , i_n)$ gives you a network with $n$ hidden layers, where $i_k$ gives you the number of neurons in the $k$th hidden layer.
If you want three hidden layers ...
oW_♦
- 6,264
8
votes
Accepted
Do transformers (e.g. BERT) have an unlimited input size?
You are right that a transformer can take in an arbitrary amount of tokens even with fixed parameters, excluding the positional embedding matrix, whose size directly grows with the maximum allowed ...
- 23.8k
7
votes
Accepted
How to choose the random seed?
TL:DR, I would suggest not to optimise over the random seed. A better investment of the time would be to improve other parts of your model, such as the pipeline, the underlying algorithms, the loss ...
- 14.7k
7
votes
What is the most efficient method for hyperparameter optimization in scikit-learn?
Optimization isn't my field, but as far as I know, efficient and effective hyper-parameter optimization these days heavily revolves around building a surrogate model. As models increase in complexity, ...
- 371
7
votes
Accepted
ROC AUC score is much less than average cross validation score
Your test score is incorrect. The ROC curve needs the probability scores from the model, not the class decisions. So replace
...
- 11.2k
7
votes
Accepted
XGBoost and Random Forest: ntrees vs. number of boosting rounds vs. n_estimators
As I understand it, iterations is equivalent to boosting rounds.
However, number of trees is not necessarily equivalent to the above, as xgboost has a parameter ...
- 1,567
6
votes
Accepted
How do scientists come up with the correct Hidden Markov Model parameters and topology to use?
I'm familiar with three main approaches:
A priori. You might know that there are four base pairs to pick from, and so allow the HMM to have four states. Or you might know that English has 44 phonemes,...
- 735
6
votes
Choosing a learning rate
Learning rate , transformed as "step size" during our iteration process , has been a hot issue for years , and it will go on .
There are three options for step size in my concerning :
One is related ...
- 399
6
votes
Which parameters are hyper parameters in a linear regression?
Hyper-parameters by definition are input parameters which are necessarily required by an algorithm to learn from data.
For standard linear regression i.e OLS, there is none. The number/ choice of ...
- 820
6
votes
Is it OK to try to find the best PCA k parameter as we do with other hyperparameters?
Your emphasis on using a validation set rather than the training set for selecting $k$ is a good practice and should be followed. However, we can do even better!
The parameter $k$ in $\text{PCA}$ is ...
- 9,247
6
votes
XGBoost and Random Forest: ntrees vs. number of boosting rounds vs. n_estimators
Following from your comment...
Why would I want to create multiple trees per iteration?
Multiple trees per iteration may mean that the model's prediction power improves much faster than using a ...
- 1,567
5
votes
What is the difference between model hyperparameters and model parameters?
In simplified words,
Model Parameters are something that a model learns on its own.
For example,
1) Weights or Coefficients of independent variables in Linear regression model.
2) Weights or ...
5
votes
Choosing a learning rate
Adding to David's answer, in fastai is where I found the concept of finding the best learning rate for that data, using a particular architecture.
But that thing exists only on fastai/pytorch. ...
- 626
5
votes
Hyperparameter search for LSTM-RNN using Keras (Python)
I would recommend Bayesian Optimization for hyper parameter search and had good results with Spearmint. You might have to use an older version for commercial use.
- 71
5
votes
Hyperparameter search for LSTM-RNN using Keras (Python)
An embedding layer turns positive integers (indexes) into dense vectors of fixed size. For instance, [[4], [20]] -> [[0.25, 0.1], [0.6, -0.2]]. This ...
- 790
5
votes
How to set the number of neurons and layers in neural networks
Working with neural networks since two years ago, this is a problem I always have each time I wan't to model a new system. The best approach I've found is the following:
Look for similar problems ...
- 760
5
votes
How to set hyperparameters in SVM classification
You could use cross validation with grid search as shown here
5
votes
Accepted
Grid search or gradient descent?
The main distinction to be made here is between a parameter and a hyperparameter; once we have this clarified, the rest is easy: grid search is not used for tuning the parameters (only the ...
- 1,958
4
votes
Accepted
How can the process of hypertuning of XGBoost parameters be automated?
In general, if you want to automate fine tuning a model's hyper parameters, its best to use a well tested package such as caret or MLR.
I've used the caret package extensively. Here is a reference of ...
- 2,517
Only top scored, non community-wiki answers of a minimum length are eligible