36 votes
Accepted

What is the difference between model hyperparameters and model parameters?

Hyperparameters and parameters are often used interchangeably but there is a difference between them. You can call something a 'hyperparameter' if it cannot be learned within the estimator directly. ...
  • 2,961
32 votes

What is the difference between model hyperparameters and model parameters?

In addition to the answer above. Model parameters are the properties of the training data that are learnt during training by the classifier or other ml model. For example in case of some NLP task: ...
  • 2,057
27 votes

How to set the number of neurons and layers in neural networks

The consideration of the number of neurons for each layer and number of layers in fully connected networks depends on the feature space of the problem. For illustrating what happens in the two ...
  • 13.5k
16 votes

How to set the number of neurons and layers in neural networks

Very good question, as there doesn't exist an exact answer to this question yet. This is an active field of research. Ultimately, the architecture of your network is related to the dimensionality of ...
15 votes

What is the difference between model hyperparameters and model parameters?

Hyper-parameters are those which we supply to the model, for example: number of hidden Nodes and Layers,input features, Learning Rate, Activation Function etc in Neural Network, while Parameters are ...
12 votes

Choosing a learning rate

Selecting a learning rate is an example of a "meta-problem" known as hyperparameter optimization. The best learning rate depends on the problem at hand, as well as on the architecture of the model ...
  • 558
12 votes

How to set the number of neurons and layers in neural networks

Short Answer: It is very related to the dimensions of your data and the type of the application. Choosing the right number of layers can only be achievable with practice. There is no general answer ...
  • 1,185
10 votes

Choosing a learning rate

Copy-pasted from my masters thesis: If the loss does not decrease for several epochs, the learning rate might be too low. The optimization process might also be stuck in a local minimum. Loss being ...
  • 17.9k
9 votes

How can you decide the window size on a pooling layer?

When you get a bit more insight into network topologies these hyperparameters will make more sense, but in general this is just like any other hyperparameter, you will have to test some settings and ...
9 votes
Accepted

Which is first ? Tuning the parameters or selecting the model

You can tune parameters only if you have already trained the model, otherwise there is nothing to tune. However, i've also read that model selection shoud be done before tuning the parameters. ...
8 votes

What is the difference between model hyperparameters and model parameters?

In machine learning, a model $M$ with parameters and hyper-parameters looks like, $Y \approx M_{\mathcal{H}}(\Phi | D)$ where $\Phi$ are parameters and $\mathcal{H}$ are hyper-parameters. $D$ is ...
8 votes
Accepted

How to implement Python's MLPClassifier with gridsearchCV?

A tuple of the form $(i_1, i_2, i_3, ... , i_n)$ gives you a network with $n$ hidden layers, where $i_k$ gives you the number of neurons in the $k$th hidden layer. If you want three hidden layers ...
  • 6,035
8 votes
Accepted

Which parameters are hyper parameters in a linear regression?

I like the way Wikipedia generally defines it: In machine learning, a hyperparameter is a parameter whose value is set before the learning process begins. By contrast, the values of other ...
  • 4,046
7 votes

What is the most efficient method for hyperparameter optimization in scikit-learn?

Optimization isn't my field, but as far as I know, efficient and effective hyper-parameter optimization these days heavily revolves around building a surrogate model. As models increase in complexity, ...
  • 361
7 votes
Accepted

ROC AUC score is much less than average cross validation score

Your test score is incorrect. The ROC curve needs the probability scores from the model, not the class decisions. So replace ...
  • 9,982
6 votes

Choosing a learning rate

Learning rate , transformed as "step size" during our iteration process , has been a hot issue for years , and it will go on . There are three options for step size in my concerning : One is related ...
  • 399
6 votes
Accepted

How do scientists come up with the correct Hidden Markov Model parameters and topology to use?

I'm familiar with three main approaches: A priori. You might know that there are four base pairs to pick from, and so allow the HMM to have four states. Or you might know that English has 44 phonemes,...
6 votes
Accepted

How to choose the random seed?

TL:DR, I would suggest not to optimise over the random seed. A better investment of the time would be to improve other parts of your model, such as the pipeline, the underlying algorithms, the loss ...
  • 14.1k
6 votes
Accepted

XGBoost and Random Forest: ntrees vs. number of boosting rounds vs. n_estimators

As I understand it, iterations is equivalent to boosting rounds. However, number of trees is not necessarily equivalent to the above, as xgboost has a parameter ...
  • 1,517
5 votes

What is the difference between model hyperparameters and model parameters?

In simplified words, Model Parameters are something that a model learns on its own. For example, 1) Weights or Coefficients of independent variables in Linear regression model. 2) Weights or ...
5 votes

Hyperparameter search for LSTM-RNN using Keras (Python)

I would recommend Bayesian Optimization for hyper parameter search and had good results with Spearmint. You might have to use an older version for commercial use.
5 votes

Hyperparameter search for LSTM-RNN using Keras (Python)

An embedding layer turns positive integers (indexes) into dense vectors of fixed size. For instance, [[4], [20]] -> [[0.25, 0.1], [0.6, -0.2]]. This ...
  • 790
5 votes

Choosing a learning rate

Adding to David's answer, in fastai is where I found the concept of finding the best learning rate for that data, using a particular architecture. But that thing exists only on fastai/pytorch. ...
  • 626
5 votes

How to set the number of neurons and layers in neural networks

Working with neural networks since two years ago, this is a problem I always have each time I wan't to model a new system. The best approach I've found is the following: Look for similar problems ...
5 votes

Which parameters are hyper parameters in a linear regression?

Hyper-parameters by definition are input parameters which are necessarily required by an algorithm to learn from data. For standard linear regression i.e OLS, there is none. The number/ choice of ...
5 votes

How to set hyperparameters in SVM classification

You could use cross validation with grid search as shown here
5 votes

Is it OK to try to find the best PCA k parameter as we do with other hyperparameters?

Your emphasis on using a validation set rather than the training set for selecting $k$ is a good practice and should be followed. However, we can do even better! The parameter $k$ in $\text{PCA}$ is ...
  • 8,667
5 votes
Accepted

Grid search or gradient descent?

The main distinction to be made here is between a parameter and a hyperparameter; once we have this clarified, the rest is easy: grid search is not used for tuning the parameters (only the ...
  • 1,820
5 votes

XGBoost and Random Forest: ntrees vs. number of boosting rounds vs. n_estimators

Following from your comment... Why would I want to create multiple trees per iteration? Multiple trees per iteration may mean that the model's prediction power improves much faster than using a ...
  • 1,517
4 votes

Choosing a learning rate

Neural networks are often trained by gradient descent on the weights. This means at each iteration we use backpropagation to calculate the derivative of the loss function with respect to each weight ...

Only top scored, non community-wiki answers of a minimum length are eligible