2022 Developer Survey is open! Take survey.
85 votes
Accepted

Choosing a learning rate

Is the learning rate related to the shape of the error gradient, as it dictates the rate of descent? In plain SGD, the answer is no. A global learning rate is used which is indifferent to the error ...
user avatar
  • 4,149
36 votes
Accepted

What is the difference between model hyperparameters and model parameters?

Hyperparameters and parameters are often used interchangeably but there is a difference between them. You can call something a 'hyperparameter' if it cannot be learned within the estimator directly. ...
user avatar
  • 2,911
32 votes

What is the difference between model hyperparameters and model parameters?

In addition to the answer above. Model parameters are the properties of the training data that are learnt during training by the classifier or other ml model. For example in case of some NLP task: ...
user avatar
  • 1,977
27 votes

How to set the number of neurons and layers in neural networks

The consideration of the number of neurons for each layer and number of layers in fully connected networks depends on the feature space of the problem. For illustrating what happens in the two ...
user avatar
  • 13.3k
26 votes

Choosing a learning rate

Below is a very good note (page 12) on learning rate in Neural Nets (Back Propagation) by Andrew Ng. You will find details relating to learning rate. http://web.stanford.edu/class/cs294a/...
user avatar
  • 361
16 votes

How to set the number of neurons and layers in neural networks

Very good question, as there doesn't exist an exact answer to this question yet. This is an active field of research. Ultimately, the architecture of your network is related to the dimensionality of ...
user avatar
15 votes

What is the difference between model hyperparameters and model parameters?

Hyper-parameters are those which we supply to the model, for example: number of hidden Nodes and Layers,input features, Learning Rate, Activation Function etc in Neural Network, while Parameters are ...
user avatar
12 votes

Choosing a learning rate

Selecting a learning rate is an example of a "meta-problem" known as hyperparameter optimization. The best learning rate depends on the problem at hand, as well as on the architecture of the model ...
user avatar
  • 558
12 votes

How to set the number of neurons and layers in neural networks

Short Answer: It is very related to the dimensions of your data and the type of the application. Choosing the right number of layers can only be achievable with practice. There is no general answer ...
user avatar
  • 1,165
10 votes

Choosing a learning rate

Copy-pasted from my masters thesis: If the loss does not decrease for several epochs, the learning rate might be too low. The optimization process might also be stuck in a local minimum. Loss being ...
user avatar
  • 17.4k
8 votes

What is the difference between model hyperparameters and model parameters?

In machine learning, a model $M$ with parameters and hyper-parameters looks like, $Y \approx M_{\mathcal{H}}(\Phi | D)$ where $\Phi$ are parameters and $\mathcal{H}$ are hyper-parameters. $D$ is ...
user avatar
8 votes

How can you decide the window size on a pooling layer?

When you get a bit more insight into network topologies these hyperparameters will make more sense, but in general this is just like any other hyperparameter, you will have to test some settings and ...
user avatar
8 votes
Accepted

Which is first ? Tuning the parameters or selecting the model

You can tune parameters only if you have already trained the model, otherwise there is nothing to tune. However, i've also read that model selection shoud be done before tuning the parameters. ...
user avatar
7 votes
Accepted

How to implement Python's MLPClassifier with gridsearchCV?

A tuple of the form $(i_1, i_2, i_3, ... , i_n)$ gives you a network with $n$ hidden layers, where $i_k$ gives you the number of neurons in the $k$th hidden layer. If you want three hidden layers ...
user avatar
  • 5,959
7 votes
Accepted

Which parameters are hyper parameters in a linear regression?

I like the way Wikipedia generally defines it: In machine learning, a hyperparameter is a parameter whose value is set before the learning process begins. By contrast, the values of other ...
user avatar
  • 3,936
7 votes
Accepted

ROC AUC score is much less than average cross validation score

Your test score is incorrect. The ROC curve needs the probability scores from the model, not the class decisions. So replace ...
user avatar
  • 9,303
6 votes

Choosing a learning rate

Learning rate , transformed as "step size" during our iteration process , has been a hot issue for years , and it will go on . There are three options for step size in my concerning : One is related ...
user avatar
  • 399
6 votes
Accepted

How do scientists come up with the correct Hidden Markov Model parameters and topology to use?

I'm familiar with three main approaches: A priori. You might know that there are four base pairs to pick from, and so allow the HMM to have four states. Or you might know that English has 44 phonemes,...
user avatar
6 votes
Accepted

How to choose the random seed?

TL:DR, I would suggest not to optimise over the random seed. A better investment of the time would be to improve other parts of your model, such as the pipeline, the underlying algorithms, the loss ...
user avatar
  • 13.8k
6 votes

What is the most efficient method for hyperparameter optimization in scikit-learn?

Optimization isn't my field, but as far as I know, efficient and effective hyper-parameter optimization these days heavily revolves around building a surrogate model. As models increase in complexity, ...
user avatar
  • 351
5 votes

What is the difference between model hyperparameters and model parameters?

In simplified words, Model Parameters are something that a model learns on its own. For example, 1) Weights or Coefficients of independent variables in Linear regression model. 2) Weights or ...
user avatar
5 votes

Hyperparameter search for LSTM-RNN using Keras (Python)

I would recommend Bayesian Optimization for hyper parameter search and had good results with Spearmint. You might have to use an older version for commercial use.
user avatar
5 votes

Hyperparameter search for LSTM-RNN using Keras (Python)

An embedding layer turns positive integers (indexes) into dense vectors of fixed size. For instance, [[4], [20]] -> [[0.25, 0.1], [0.6, -0.2]]. This ...
user avatar
  • 790
5 votes

Choosing a learning rate

Adding to David's answer, in fastai is where I found the concept of finding the best learning rate for that data, using a particular architecture. But that thing exists only on fastai/pytorch. ...
user avatar
  • 626
5 votes

How to set the number of neurons and layers in neural networks

Working with neural networks since two years ago, this is a problem I always have each time I wan't to model a new system. The best approach I've found is the following: Look for similar problems ...
user avatar
5 votes

Which parameters are hyper parameters in a linear regression?

Hyper-parameters by definition are input parameters which are necessarily required by an algorithm to learn from data. For standard linear regression i.e OLS, there is none. The number/ choice of ...
user avatar
5 votes

How to set hyperparameters in SVM classification

You could use cross validation with grid search as shown here
user avatar
5 votes

Is it OK to try to find the best PCA k parameter as we do with other hyperparameters?

Your emphasis on using a validation set rather than the training set for selecting $k$ is a good practice and should be followed. However, we can do even better! The parameter $k$ in $\text{PCA}$ is ...
user avatar
  • 8,499
5 votes
Accepted

XGBoost and Random Forest: ntrees vs. number of boosting rounds vs. n_estimators

As I understand it, iterations is equivalent to boosting rounds. However, number of trees is not necessarily equivalent to the above, as xgboost has a parameter ...
user avatar
  • 1,487
4 votes

Choosing a learning rate

Neural networks are often trained by gradient descent on the weights. This means at each iteration we use backpropagation to calculate the derivative of the loss function with respect to each weight ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible