Questions tagged [hyperparameter]

Hyperparameters of a model are the kind of parameters that cannot be directly learned during training but are set beforehand. Hyperparameters can define, for example, the complexity of the model or its capacity to learn.

Filter by
Sorted by
Tagged with
3
votes
1answer
1k views

Automated tuning of Hyperparameter

Are there any advanced packages that allows automated tuning of hyperparameters for neural network and traditional machine learning algorithms like XGBoost, random forest (using method like Bayesian, ...
35
votes
6answers
7k views

How to set the number of neurons and layers in neural networks

I am a beginner to neural networks and have had trouble grasping two concepts: How does one decide the number of middle layers a given neural network have? 1 vs. 10 or whatever. How does one decide ...
3
votes
1answer
588 views

Basic method of optimizing hyperparameters

I recently read the LIPO blog post on the dlib blog: http://blog.dlib.net/2017/12/a-global-optimization-algorithm-worth.html It mentions that it can be used for optimizing hyperparameters of eg ...
2
votes
1answer
62 views

How to think about prediction error that is not convex in hyperparameter, or over the course of training

Take the following case of a hyperparameter and prediction error: Imagine that the hyperparameter is a L2 penalty or a dropout rate -- something that we think that should have a single sweet spot -- ...
1
vote
4answers
4k views

Parameter Tuning by Cross Validation for Random Forest

I train a binary random forest classifier on scikit-learn's 20 newsgroups dataset. I want to tune the parameters and try so by gridsearch and 3-fold cross validation on the training data. Is there ...
2
votes
1answer
259 views

Strategies for automatically tuning the hyper-parameters of deep learning models

I'm interested in trying to develop a framework for automatically tuning the hyperparameters of deep learning models and I'm looking for advice / references to resources that may be useful and ...
1
vote
3answers
285 views

Grid Search and High Variance

I am currently trying to optimise some parameters on my model (15000 samples). What I am finding is a relatively large variance in the loss function 2%-10% which makes it hard to identify which ...
4
votes
1answer
4k views

Which is better: Out of Bag (OOB) or Cross-Validation (CV) error estimates?

I have seen other posts in this forum but didn't find any convincing answer. Random Forest has an another way of tuning hyperparameter via OOB by design. OOB and CV are not the same as OOB error is ...
0
votes
1answer
352 views

Can the learning rate be considered both a parameter AND a hyper-parameter?

Here is my understanding of those 2 terms: Hyper-parameter: A variable that is set by a human before the training process starts. Examples are the number of hidden-layers in a Neural Network, the ...
4
votes
1answer
24k views

How to implement Python's MLPClassifier with gridsearchCV?

I am trying to implement Python's MLPClassifier with 10 fold cross-validation using gridsearchCV function. Here is a chunk of my code: ...
1
vote
1answer
1k views

Bounds for MLP hyperparameter search

I'm trying to optimize a neural network architecture for a particular problem, but there just seems to be so many hyperparameters that I'm concerned that there are much better options that I'm not ...
5
votes
1answer
6k views

How can you decide the window size on a pooling layer?

On the convolutional neural network, there used one or more pooling layers. As far as I know many tutorials instruct you to set it either 2 or ...
1
vote
0answers
199 views

Experimental design - hyperparameter optimization

I'm reading a paper by Bergstra and Bengio (2012) on random search for hyperparameter optimization. I'm confused by their graph and explanation in Section 2.2: "Figure 2 illustrates the results of a ...
2
votes
1answer
915 views

h2o, different stopping metric leads to different optimal for hyperparameters

I want to choose the "optimal" hyperparameters for gbm. So I run the following code using the h2o package ...
3
votes
1answer
2k views

Efficient way to optimise hyper parameter for network with multiple inputs?

I am currently looking for a way in which a network with multiple inputs can optimise its hyper parameter... scikit-learn has gridsearch CV but Keras only supports single inputs using the scikit-...
1
vote
1answer
2k views

Is GridSearchCV computing SVC with rbf kernel and different degrees?

I'm running a GridSearchCV with a OneVsRestClasssifer using SVC as an estimator. This is the ...
0
votes
1answer
2k views

Can't reproduce results from GridSearchCV?

I am trying to find optimized n_neighbors value for KnearestClassifier using GridSearchCV. I am able to get optimized parameters but when I enter those in my classifier results don't match with ...
1
vote
0answers
235 views

How do I choose the optimal parameters for reliefF

For feature selection I use reliefF provided by matlab. The reliefF function offers a parameter k to influence its ouput, additionaly for my specific task I can also vary a window length l on which ...
1
vote
1answer
5k views

XGboost classification with very small data set

I have a general question regarding XGboost and especially the n_rounds parameter, regarding small datasets. Normally I tune the n_rounds parameters by cross-validation, but what if you have too less ...
1
vote
0answers
181 views

Global vs. local bias-variance tradeoff

In the standard example of decomposing the MSE into Bias, Variance and Irreducible error: $$MSE(x) = \left(\mathbb{E}[\hat{f}(x)] - f(x) \right)^2 + \mathbb{E}\left[\left(\hat{f}(x) - f(x)\right)^2\...
1
vote
0answers
3k views

Why is my loss so high?

I am struggling to understand why i am getting such a high loss/val_loss rate on my training. I am training a regression network. I've normalized the input data to range between -1 to 1, and left the ...
39
votes
6answers
36k views

What is the difference between model hyperparameters and model parameters?

I have noticed that such terms as model hyperparameter and model parameter have been used interchangeably on the web without prior clarification. I think this is incorrect and needs explanation. ...
7
votes
3answers
344 views

Regression model with variable number of parameters in dataset?

I work in physics. We have lots of experimental runs, with each run yielding a result, y and some parameters that should predict the result, ...
3
votes
1answer
2k views

How can the process of hypertuning of XGBoost parameters be automated?

I'm using xgboost for training a model on a data with extreme class imbalance. After referring from here. After performing grid search and some manual settings, I ...
6
votes
1answer
4k views

Overfitting for minority class after SMOTE w/ random forests

I used SMOTE to make a predictive model, with class 1 having 1800 samples and 35000+ of class 0 samples. Hence, as per SMOTE, synthetic samples were created and the random forest was trained. However,...
2
votes
1answer
300 views

Hyperparameters in Gaussian Process

My academical background is in physics and analysis (PDE's), but now I'am reading about data science. I'm reading about Gaussian Process implementation in Sci-Kit Learn I could not find a simple ...
19
votes
4answers
20k views

Hyperparameter search for LSTM-RNN using Keras (Python)

From Keras RNN Tutorial: "RNNs are tricky. Choice of batch size is important, choice of loss and optimizer is critical, etc. Some configurations won't converge." So this is more a general question ...
3
votes
2answers
599 views

Genetic Algorithm to find best parameter values of an estimaor

I am making some stochastic training ensemble classes in Python, and I want to get hyperparameters values. Grid search will take too long for moderate data sets, because in my stochastic training I ...
10
votes
2answers
1k views

How do scientists come up with the correct Hidden Markov Model parameters and topology to use?

I understand how a Hidden Markov Model is used in genomic sequences, such as finding a gene. But I don't understand how to come up with a particular Markov model. I mean, how many states should the ...
2
votes
2answers
2k views

Tune hyperparameters for cost-sensitive classification

I have an unbalanced data set with about 8% of negative examples. The goal is to minimize false negatives given a cost matrix. It seems like SVM (with radial kernel) and random forest work best. How ...
6
votes
1answer
981 views

Neural Network Golf: smallest network for a certain level of performance

I am interested in any data, publications, etc about what is the smallest neural network that can achieve a certain level of classification performance. By small I mean few parameters, not few ...
101
votes
10answers
106k views

Choosing a learning rate

I'm currently working on implementing Stochastic Gradient Descent, SGD, for neural nets using back-propagation, and while I understand its purpose I have some ...

1 2
3