Questions tagged [hyperparameter]

Hyperparameters of a model are the kind of parameters that cannot be directly learned during training but are set beforehand. Hyperparameters can define, for example, the complexity of the model or its capacity to learn.

Filter by
Sorted by
Tagged with
101
votes
10answers
106k views

Choosing a learning rate

I'm currently working on implementing Stochastic Gradient Descent, SGD, for neural nets using back-propagation, and while I understand its purpose I have some ...
39
votes
6answers
36k views

What is the difference between model hyperparameters and model parameters?

I have noticed that such terms as model hyperparameter and model parameter have been used interchangeably on the web without prior clarification. I think this is incorrect and needs explanation. ...
35
votes
6answers
7k views

How to set the number of neurons and layers in neural networks

I am a beginner to neural networks and have had trouble grasping two concepts: How does one decide the number of middle layers a given neural network have? 1 vs. 10 or whatever. How does one decide ...
19
votes
4answers
20k views

Hyperparameter search for LSTM-RNN using Keras (Python)

From Keras RNN Tutorial: "RNNs are tricky. Choice of batch size is important, choice of loss and optimizer is critical, etc. Some configurations won't converge." So this is more a general question ...
10
votes
2answers
2k views

What is the most efficient method for hyperparameter optimization in scikit-learn?

An overview of the hyperparameter optimization process in scikit-learn is here. Exhaustive grid search will find the optimal set of hyperparameters for a model. The downside is that exhaustive grid ...
10
votes
2answers
1k views

How do scientists come up with the correct Hidden Markov Model parameters and topology to use?

I understand how a Hidden Markov Model is used in genomic sequences, such as finding a gene. But I don't understand how to come up with a particular Markov model. I mean, how many states should the ...
9
votes
4answers
3k views

Which is first ? Tuning the parameters or selecting the model

I've been reading about how we split our data into 3 parts; generally, we use the validation set to help us tune the parameters and the test set to have an unbiased estimate on how well does our model ...
7
votes
1answer
885 views

Is it OK to try to find the best PCA k parameter as we do with other hyperparameters?

Principal Component Analysis (PCA) is used to reduce n-dimensional data to k-dimensional data to speed things up in machine learning. After PCA is applied, one can check how much of the variance of ...
7
votes
3answers
344 views

Regression model with variable number of parameters in dataset?

I work in physics. We have lots of experimental runs, with each run yielding a result, y and some parameters that should predict the result, ...
6
votes
1answer
4k views

Overfitting for minority class after SMOTE w/ random forests

I used SMOTE to make a predictive model, with class 1 having 1800 samples and 35000+ of class 0 samples. Hence, as per SMOTE, synthetic samples were created and the random forest was trained. However,...
6
votes
1answer
982 views

Neural Network Golf: smallest network for a certain level of performance

I am interested in any data, publications, etc about what is the smallest neural network that can achieve a certain level of classification performance. By small I mean few parameters, not few ...
5
votes
1answer
6k views

How can you decide the window size on a pooling layer?

On the convolutional neural network, there used one or more pooling layers. As far as I know many tutorials instruct you to set it either 2 or ...
5
votes
1answer
154 views

How to make it possible for a neural network to tune its own hyper parameters?

I am curious about what would happen to hyperparameters when they would be set by a neural network itself or by creating a neural network that encapsulates and influences the hyperparameters of the ...
5
votes
1answer
68 views

Search for hyperparameters whith different features using Random Forest

I have a dataset in which I would like to perform a classification model, so I have decided to use Random Forest. The number of features that I have is approximately 200 and I would like to test which ...
4
votes
3answers
22k views

Which parameters are hyper parameters in a linear regression?

Can the number of features used in a linear regression be regarded as a hyperparameter? Perhaps the choice of features?
4
votes
1answer
356 views

ROC AUC score is much less than average cross validation score

Using Lending club Dataset to find the propability of default. I am using hyperopt library to fine tune hyper parameter for an XGBclassifier and trying to maximize the ROC AUC score. I am also using ...
4
votes
2answers
4k views

How to choose the random seed?

I understand this question can be strange, but how do I pick the final random_seed for my classifier? Below is an example code. It uses the ...
4
votes
1answer
24k views

How to implement Python's MLPClassifier with gridsearchCV?

I am trying to implement Python's MLPClassifier with 10 fold cross-validation using gridsearchCV function. Here is a chunk of my code: ...
4
votes
1answer
4k views

Which is better: Out of Bag (OOB) or Cross-Validation (CV) error estimates?

I have seen other posts in this forum but didn't find any convincing answer. Random Forest has an another way of tuning hyperparameter via OOB by design. OOB and CV are not the same as OOB error is ...
4
votes
1answer
2k views

What does the “dual” parameter in sklearn.svm.LinearSVC and sklearn.svm.LinearSVR do?

While I am more or less familiar with the idea of the SVM, I do not understand the meaning of the dual parameter, which is described in the documentation as: ...
4
votes
1answer
229 views

Hyperopt vs Default Values

When I use the hyperopt library to tune my Random Forest classifier, I get the following results: Hyperopt estimated optimum {'max_depth': 10.0, 'n_estimators': 300.0} However, when I train the ...
4
votes
1answer
460 views

Why do we need the hyperparameters beta and alpha in LDA?

I'm trying to understand the technical part of Latent Dirichlet Allocation (LDA), but I have a few questions on my mind: First: Why do we need to add alpha and gamma every time we sample the equation ...
4
votes
1answer
281 views

How many epochs to run during hyperparameter search?

If I'm doing a hyperparameter search and comparing two different hyperparameters (but not number of epochs), is there some established rule of thumb for how many epochs to run? If I just compare ...
4
votes
2answers
333 views

xgboost or lightgbm to handle Binomial problems [duplicate]

I have a dataset containing a column of trials, a column of successes and other features; and, obviously, I can generate a probability column. I would like to use gradient boosting methods (like ...
3
votes
2answers
96 views

How to set hyperparameters in SVM classification

I am studying image classification using SVMs and it is generally defined as so... N = number of training examples W = is the weights f(x, W) = dot product λ is explained to be set through cross-...
3
votes
2answers
540 views

Setting best SVM hyper parameters

I have a non linear data set, and I am using SVM (RBF kernel) to build a classification model, but not sure how to set the best hyperparameters of the SVM, C and gamma in Matlab ...
3
votes
2answers
90 views

Comparison of machine learning approaches for a topic in a scientific paper

As part of my master's thesis, I have made a prediction of data with approaches of machine learning in a topic where are no papers yet. The topic is a regression problem for which several machine ...
3
votes
1answer
2k views

How can the process of hypertuning of XGBoost parameters be automated?

I'm using xgboost for training a model on a data with extreme class imbalance. After referring from here. After performing grid search and some manual settings, I ...
3
votes
2answers
795 views

XGBoost and Random Forest: ntrees vs. number of boosting rounds vs. n_estimators

So I understand the main difference between Random Forests and GB Methods. Random Forests grow parallel trees and GB Methods grow one tree for each iteration. However, I am confused on the vocab used ...
3
votes
2answers
513 views

Shuffle the data before splitting into folds

I am running a 4-folds cross validation hyperparameter tuning using sklearn's 'cross_validate' and 'KFold' functions. Assuming that my training dataset is already shuffled, then should I for each ...
3
votes
1answer
1k views

Grid search or gradient descent?

Assume we have a neural network and one if its activation functions is a function of parameter a. We want to find the weights and parameter a that leads to the minimum loss on the validation set which ...
3
votes
1answer
493 views

Hyper parameters and ValidationSet

Please correct me if I am wrong. "Training Set is used for calculating parameters of a machine learning model, Validation data is used for calculating hyperparameters of the same model (we use same ...
3
votes
1answer
1k views

Automated tuning of Hyperparameter

Are there any advanced packages that allows automated tuning of hyperparameters for neural network and traditional machine learning algorithms like XGBoost, random forest (using method like Bayesian, ...
3
votes
2answers
599 views

Genetic Algorithm to find best parameter values of an estimaor

I am making some stochastic training ensemble classes in Python, and I want to get hyperparameters values. Grid search will take too long for moderate data sets, because in my stochastic training I ...
3
votes
1answer
589 views

Basic method of optimizing hyperparameters

I recently read the LIPO blog post on the dlib blog: http://blog.dlib.net/2017/12/a-global-optimization-algorithm-worth.html It mentions that it can be used for optimizing hyperparameters of eg ...
3
votes
1answer
822 views

Small number of estimators in gradient boosting

I am tuning a regression gradient boosting-based model to determine the appropriate hyperparameters using 4-folds cross validation. More specifically, I am using XGBoost and lightGBM for the models ...
3
votes
1answer
19 views

Do i need to use hyperparamters from Gridsearch to train on WHOLE training set to get final model?

I just want to make sure i am on the right lines so please correct me if wrong. I am testing which hyperparmets are best for logisitic regession on my data X, y where X is featrues and y is target. X, ...
3
votes
1answer
419 views

Hyperparameter tuning and cross validation

I have some confusion about proper usage of cross-validation to tune hyperparameters and evaluate estimator performance and generalizeability. As I understand it, this would be the process you would ...
3
votes
1answer
2k views

Hyperparameter tuning for stacked models

I'm reading the following kaggle post for learning how to incorporate model stacking http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/ in ML models. The structure ...
3
votes
1answer
2k views

Efficient way to optimise hyper parameter for network with multiple inputs?

I am currently looking for a way in which a network with multiple inputs can optimise its hyper parameter... scikit-learn has gridsearch CV but Keras only supports single inputs using the scikit-...
3
votes
1answer
199 views

Benefits of using Deep Learning-specific hyperparameter optimization tools vs. sklearn?

There are quite a few library for hyperparameter optimization that are specific to Keras or other Deep Learning libraries, like Hyperas or Talos. My question is, what's the main benefit of using ...
2
votes
4answers
1k views

Is it better to optimize hyperparameters or run multiple epochs?

Whenever I train a neural network I only have it go through a few epochs ( 1 to 3). This is because I am training them on a bad CPU and it would take some time to have the neural network go though ...
2
votes
2answers
98 views

Hyperparameter optimization, ensembling instead of selecting with CV criteria

While burning CPUs performing a CV selection on a thin grid put on some hyperparameter space. I am using the `scikit-learn' API, for which the end result is a single point on the hyperparameter space, ...
2
votes
1answer
917 views

h2o, different stopping metric leads to different optimal for hyperparameters

I want to choose the "optimal" hyperparameters for gbm. So I run the following code using the h2o package ...
2
votes
2answers
71 views

Faster grid search with small dataset to derive best params instead of full dataset?

I have a dataset of 300 000 rows and an ensemble model, which include grid search to find the best params of every algorithm. Unfortunately the grid search needs to long and I have problems to ...
2
votes
2answers
964 views

Why SVM gridsearch takes longer time?

I have a dataset of 5K records and 60 features focussed on binary classification. Please find my code below for SVM paramter tuning. It's running for a longer time than ...
2
votes
1answer
62 views

How to think about prediction error that is not convex in hyperparameter, or over the course of training

Take the following case of a hyperparameter and prediction error: Imagine that the hyperparameter is a L2 penalty or a dropout rate -- something that we think that should have a single sweet spot -- ...
2
votes
1answer
272 views

Hyperparameter optimization performance comparison

I have used Bayesian optimization for hyperparameter tuning in a machine learning model. What is the best way to compare the performance of network with and without Bayesian optimization? I found some ...
2
votes
1answer
264 views

How to handle the parameter space of neural networks?

This question is very broad (and might even be closed as "too broad"). It can be considered as a beginners question, because it is largely about getting started in terms of heading into a direction ...
2
votes
1answer
228 views

Do we need to increase training data size when increasing dropouts?

I am using a fully connected feed forward neural network built using keras for text classification. It consists of 3 hidden layer. I am planning to add a dropout layer after each hidden layer to ...