77

Is the learning rate related to the shape of the error gradient, as it dictates the rate of descent? In plain SGD, the answer is no. A global learning rate is used which is indifferent to the error gradient. However, the intuition you are getting at has inspired various modifications of the SGD update rule. If so, how do you use this information to inform ...


32

Hyperparameters and parameters are often used interchangeably but there is a difference between them. You call something a 'hyperparameter' if it cannot be learned within the estimator directly. However, 'parameters' is more general term. When you say 'passing the parameters to the model', it generally means a combination of hyperparameters along with some ...


26

In addition to the answer above. Model parameters are the properties of the training data that are learnt during training by the classifier or other ml model. For example in case of some NLP task: word frequency, sentence length, noun or verb distribution per sentence, the number of specific character n-grams per word, lexical diversity, etc. Model ...


23

The consideration of the number of neurons for each layer and number of layers in fully connected networks depends on the feature space of the problem. For illustrating what happens in the two dimensional cases in order to depict, I use 2-d space. I have used images from the works of a scientist. For understanding other nets like CNN I recommend you taking a ...


22

Below is a very good note (page 12) on learning rate in Neural Nets (Back Propagation) by Andrew Ng. You will find details relating to learning rate. http://web.stanford.edu/class/cs294a/sparseAutoencoder_2011new.pdf For your 4th point, you're right that normally one has to choose a "balanced" learning rate, that should neither overshoot nor converge too ...


13

Very good question, as there doesn't exist an exact answer to this question yet. This is an active field of research. Ultimately, the architecture of your network is related to the dimensionality of your data. Since neural networks are universal approximators, as long as your network is big enough, it has the ability to fit your data. The only way to truly ...


10

Hyper-parameters are those which we supply to the model, for example: number of hidden Nodes and Layers,input features, Learning Rate, Activation Function etc in Neural Network, while Parameters are those which would be learned by the machine like Weights and Biases.


10

Selecting a learning rate is an example of a "meta-problem" known as hyperparameter optimization. The best learning rate depends on the problem at hand, as well as on the architecture of the model being optimized, and even on the state of the model in the current optimization process! There are even software packages devoted to hyperparameter optimization ...


9

Short Answer: It is very related to the dimensions of your data and the type of the application. Choosing the right number of layers can only be achievable with practice. There is no general answer to this question yet. By choosing a network architecture, you constrain your space of possibilities (hypothesis space) to a specific series of tensor operations, ...


8

Copy-pasted from my masters thesis: If the loss does not decrease for several epochs, the learning rate might be too low. The optimization process might also be stuck in a local minimum. Loss being NAN might be due to too high learning rates. Another reason is division by zero or taking the logarithm of zero. Weight update tracking: Andrej Karpathy proposed ...


7

When you get a bit more insight into network topologies these hyperparameters will make more sense, but in general this is just like any other hyperparameter, you will have to test some settings and see what works better. In the case of pooling layers it is actually relatively interpretable. Why do we use pooling? To downsample our feature maps. This is done ...


7

You can tune parameters only if you have already trained the model, otherwise there is nothing to tune. However, i've also read that model selection shoud be done before tuning the parameters. Before tuning you need to do some kind of pre-processing before tuning the parameters. Usually your pipeline will consist of: Get Data and Clean It. Do some EDA ( ...


7

Your test score is incorrect. The ROC curve needs the probability scores from the model, not the class decisions. So replace y_predicted = grid_clf.predict(X_test) with y_predicted = grid_clf.predict_proba(X_test)[:,1]


6

In machine learning, a model $M$ with parameters and hyper-parameters looks like, $Y \approx M_{\mathcal{H}}(\Phi | D)$ where $\Phi$ are parameters and $\mathcal{H}$ are hyper-parameters. $D$ is training data and $Y$ is output data (class labels in case of classification task). The objective during training is to find estimate of parameters $\hat{\Phi}$ ...


6

I'm familiar with three main approaches: A priori. You might know that there are four base pairs to pick from, and so allow the HMM to have four states. Or you might know that English has 44 phonemes, and so have 44 states for the hidden phoneme layer in a voice recognition model. Estimation. The number of states can often be estimated beforehand, perhaps ...


6

TL:DR, I would suggest not to optimise over the random seed. A better investment of the time would be to improve other parts of your model, such as the pipeline, the underlying algorithms, the loss function... heck, even optimise the runtime performance! :-) This is an interesting question, even though (in my opinion) should not be a parameter to optimise. ...


6

Optimization isn't my field, but as far as I know, efficient and effective hyper-parameter optimization these days heavily revolves around building a surrogate model. As models increase in complexity, they become a more opaque black box. This is the case for deep neural nets and presumably complex trees as well. A surrogate model attempts to regress the ...


5

I would recommend Bayesian Optimization for hyper parameter search and had good results with Spearmint. You might have to use an older version for commercial use.


5

An embedding layer turns positive integers (indexes) into dense vectors of fixed size. For instance, [[4], [20]] -> [[0.25, 0.1], [0.6, -0.2]]. This representation conversion is learned automatically with the embedding layer in Keras (see the documentation). However, it seems that your data does not need any such embedding layer to perform a conversion. ...


5

Learning rate , transformed as "step size" during our iteration process , has been a hot issue for years , and it will go on . There are three options for step size in my concerning : One is related to "time" , and each dimension shall share the same step size . You might have noticed something like $\it\huge\bf\frac{\alpha}{\sqrt{t}}$ while t ...


5

A tuple of the form $(i_1, i_2, i_3, ... , i_n)$ gives you a network with $n$ hidden layers, where $i_k$ gives you the number of neurons in the $k$th hidden layer. If you want three hidden layers with $10,30$ and $20$ neurons, your tuple would need to look like $(10,30,20)$. $(100,1)$ would mean that the second hidden layer only has one neuron.


5

I like the way Wikipedia generally defines it: In machine learning, a hyperparameter is a parameter whose value is set before the learning process begins. By contrast, the values of other parameters are derived via training. On top of what Wikipedia says I would add: Hyperparameter is a parameter that concerns the numerical optimization problem at ...


5

You could use cross validation with grid search as shown here


5

Your emphasis on using a validation set rather than the training set for selecting $k$ is a good practice and should be followed. However, we can do even better! The parameter $k$ in $\text{PCA}$ is more special than a general hyper-parameter. Because, the solution to $\text{PCA}(k)$ already exists in $\text{PCA}(K)$, for $K > k$, which is the first $k$ ...


4

Adding to David's answer, in fastai is where I found the concept of finding the best learning rate for that data, using a particular architecture. But that thing exists only on fastai/pytorch. Recently someone made a keras implementation. which in turn are based on these papers: A disciplined approach to neural network hyper-parameters: Part 1 -- learning ...


4

In simplified words, Model Parameters are something that a model learns on its own. For example, 1) Weights or Coefficients of independent variables in Linear regression model. 2) Weights or Coefficients of independent variables SVM. 3) Split points in Decision Tree. Model hyper-parameters are used to optimize the model performance. For example, 1)...


4

In general, if you want to automate fine tuning a model's hyper parameters, its best to use a well tested package such as caret or MLR. I've used the caret package extensively. Here is a reference of the parameters supported by caret for tuning a xgboost model. To automatically select parameters using caret, do the following: First define a range of ...


4

First of all, you are using different metrics to determine how well you are doing, that means it's not weird that different metrics find different hyperparameter settings that work better. Second of all, some hyperparameters might not matter for the problem you are solving, which means all the signal you are getting from those hyperparameters is noise. Third ...


4

OOB samples are a very efficient way to obtain error estimates for random forests. From a computational perspective, OOB are definitely preferred over CV. Also, it holds that if the number of bootstrap samples is large enough, CV and OOB samples will produce the same (or very similar) error estimates. Thus, if you perform many bootstrap samples, I would ...


4

Working with neural networks since two years ago, this is a problem I always have each time I wan't to model a new system. The best approach I've found is the following: Look for similar problems that have also been modeled with feed-forward networks and study their architectures. Begin with that configuration, train the data set and evaluate the test set. ...


Only top voted, non community-wiki answers of a minimum length are eligible