Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [hyperparameter]

Hyperparameters of a model are the kind of parameters that cannot be directly learned during training but are set beforehand. Hyperparameters can define, for example, the complexity of the model or its capacity to learn.

0
votes
0answers
3 views

Hyper-parameter tuning when you don't have an access to the test data

I'm building models for SQUAD (Stanford Question Answering) dataset (https://rajpurkar.github.io/SQuAD-explorer). Stanford doesn't release its test set. It only provides us with training and dev ...
0
votes
0answers
14 views

Choosing the right hyperparameter and score for building ensemble

I want to build an ensemble model from individual classifiers(e.g KNN,SVM etc) for classification purpose. Before building the ensemble mode, I want to select the best hyperparameter from the ...
5
votes
4answers
115 views

Which is first ? Tuning the parameters or selecting the model

I've been reading about how we split our data into 3 parts; generally, we use the validation set to help us tune the parameters and the test set to have an unbiased estimate on how well does our model ...
3
votes
1answer
32 views

Hyperparameter tuning for stacked models

I'm reading the following kaggle post for learning how to incorporate model stacking http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/ in ML models. The structure ...
1
vote
0answers
13 views

How to tune parameters for Time Series Analysis, when forecasting is only dominated by one feature and error is not getting reduced?

I am trying to predict time series based on 150 features. When I plot correlation of these features, I am getting 20 features with more or less importance but every model I use, it is completely ...
1
vote
0answers
12 views

How to optimize the separation of two distributions from binary classfication

Given a sample where for each individual a classification is predetermined (e.g. sick or not) and 5 random variables are measured. The random variables are on the same scale but from differnt bins. E....
1
vote
0answers
12 views

std::bad_alloc with densenet and hyperas

I am filing this issue after being stagnated here for couple of weeks. I am using hyperas to find the hyperparameters for my network, Densenet. My issue here is that my evaluation always fails with ...
0
votes
1answer
31 views

Denoising Autoencoder Parameter Search

I have ran a hyperparameter search for a denoising autoencoder and the results suggest I should make the sizes of my hidden layers as large as possible (within the range of values I allowed it to ...
1
vote
1answer
43 views

What is difference between final episodes of training and test in DQN?

What is difference between running in final episode of training mode and running in test mode in DQN? Is there any difference more than after training and tune the hyper-parameters, we test for one ...
0
votes
0answers
13 views

Relationship between number of free parameters and number of samples required to train a graphical model

Is there a rule of thumb for how many data samples you should have to train a graphical model with N free parameters? In my case I am using a hidden Markov model with Gaussian emissions, which for K ...
0
votes
0answers
17 views

How to interpret a max_depth very small of my xgboost classifier

I have a binary classification problem. I optimize max_depth and min_child_weight by GridSearchCV (nfold = 5, fold size ~500). The parameter range varies for max_depth from 2 to 30 and for ...
3
votes
1answer
46 views

How to make it possible for a neural network to tune its own hyper parameters?

I am curious about what would happen to hyperparameters when they would be set by a neural network itself or by creating a neural network that encapsulates and influences the hyperparameters of the ...
0
votes
0answers
28 views

How to perform platt scaling for hyperparameter-optimized model?

I'm using Python and have a best estimator from a grid search. Wanted to be able to calibrate the probability output accordingly, but would like to know more about implementing platt scaling. From ...
2
votes
2answers
260 views

How to choose the random seed?

I understand this question can be strang, but how do I pick the final random_seed for my classifier? Below is an example code. It uses the ...
0
votes
1answer
78 views

Benefits of using Deep Learning-specific hyperparameter optimization tools vs. sklearn?

There are quite a few library for hyperparameter optimization that are specific to Keras or other Deep Learning libraries, like Hyperas or Talos. My question is, what's the main benefit of using ...
0
votes
0answers
98 views

Two different approaches of oversampling data with GridSearchCV leads to similar test results

I was trying to compare two approaches to optimal selection of hyperparameters based on two approaches: 1) Wrong Approach: Oversampling before GridSearch CV This can lead to bleeding of data (that ...
2
votes
4answers
108 views

Is it better to optimize hyperparameters or run multiple epochs?

Whenever I train a neural network I only have it go through a few epochs ( 1 to 3). This is because I am training them on a bad CPU and it would take some time to have the neural network go though ...
0
votes
0answers
17 views

Difference between MOE and Spearmint?

MOE seems to be a plain Bayesian optimization. Just curious if anyone knows about the difference.
0
votes
1answer
71 views

Random Forest - Explanation Parameter

I got some question about the "standard" parameter from a random forest. Following I write my understanding about these parameters. I would be glad if I could confirm my understanding or correct it. :)...
3
votes
1answer
62 views

Hyper parameters and ValidationSet

Please correct me if I am wrong. "Training Set is used for calculating parameters of a machine learning model, Validation data is used for calculating hyperparameters of the same model (we use same ...
4
votes
1answer
220 views

What does the “dual” parameter in sklearn.svm.LinearSVC and sklearn.svm.LinearSVR do?

While I am more or less familiar with the idea of the SVM, I do not understand the meaning of the dual parameter, which is described in the documentation as: ...
1
vote
2answers
79 views

How to tune bandwidth in machine learning kernel model?

Gaussian kernel $k(x,y) = \exp(-\lVert x-y \rVert^2/\sigma^2)$ has a hyperparameter $\sigma$. I know grid search cross validation, but this would require a lot of computation since computational ...
1
vote
2answers
2k views

Which parameters are hyper parameters in a linear regression?

Can the number of features used in a linear regression be regarded as a hyperparameter? Perhaps the choice of features?
0
votes
1answer
185 views

Is there any alternative to L-BFGS-B algorithm for hyperparameter optimization in Scikit learn?

The Gaussian process regression can be computed in scikit learn using an object of class GaussianProcessRegressor as: ...
1
vote
1answer
372 views

Optimising Kernel parameters using training data in GaussianProcessRegressor of Scikit-learn

I want to optimize the Kernel parameters or hyper-parameters using my training data in GaussianProcessRegressor of Scikit-learn.Following is my query: My training datasets are: X: 2-D Cartesian ...
3
votes
1answer
133 views

Why do we need the hyperparameters beta and alpha in LDA?

I'm trying to understand the technical part of Latent Dirichlet Allocation (LDA), but I have a few questions on my mind: First: Why do we need to add alpha and gamma every time we sample the equation ...
0
votes
0answers
13 views

Does selecting metrics while modelling plays a role?

Bear with me if its a noobish question, Today itself i was trying to fit a model using simple Logistic Regression (LR) followed by a ...
2
votes
2answers
142 views

Do we need to increase training data size when increasing dropouts?

I am using a fully connected feed forward neural network built using keras for text classification. It consists of 3 hidden layer. I am planning to add a dropout layer after each hidden layer to ...
1
vote
0answers
119 views

How many epochs to run during hyperparameter search?

If I'm doing a hyperparameter search and comparing two different hyperparameters (but not number of epochs), is there some established rule of thumb for how many epochs to run? If I just compare ...
2
votes
2answers
78 views

Comparison of machine learning approaches for a topic in a scientific paper

As part of my master's thesis, I have made a prediction of data with approaches of machine learning in a topic where are no papers yet. The topic is a regression problem for which several machine ...
2
votes
1answer
155 views

Parameter tuning for machine learning algorithms

When it comes to the topic of tuning parameters, most of the time you read grid search. But if you have 6 parameters, for which you want to test 10 variants, you get to 10^6 = 1000000 runs. Which in ...
3
votes
1answer
558 views

Automated tuning of Hyperparameter

Are there any advanced packages that allows automated tuning of hyperparameters for neural network and traditional machine learning algorithms like XGBoost, random forest (using method like Bayesian, ...
14
votes
4answers
1k views

How to set the number of neurons and layers in neural networks

I am a beginner to neural networks and have had trouble grasping two concepts: How does one decide the number of middle layers a given neural network have? 1 vs. 10 or whatever. How does one decide ...
3
votes
1answer
148 views

Basic method of optimizing hyperparameters

I recently read the LIPO blog post on the dlib blog: http://blog.dlib.net/2017/12/a-global-optimization-algorithm-worth.html It mentions that it can be used for optimizing hyperparameters of eg ...
2
votes
1answer
44 views

How to think about prediction error that is not convex in hyperparameter, or over the course of training

Take the following case of a hyperparameter and prediction error: Imagine that the hyperparameter is a L2 penalty or a dropout rate -- something that we think that should have a single sweet spot -- ...
1
vote
4answers
2k views

Parameter Tuning by Cross Validation for Random Forest

I train a binary random forest classifier on scikit-learn's 20 newsgroups dataset. I want to tune the parameters and try so by gridsearch and 3-fold cross validation on the training data. Is there ...
2
votes
1answer
185 views

Strategies for automatically tuning the hyper-parameters of deep learning models

I'm interested in trying to develop a framework for automatically tuning the hyperparameters of deep learning models and I'm looking for advice / references to resources that may be useful and ...
1
vote
3answers
96 views

Grid Search and High Variance

I am currently trying to optimise some parameters on my model (15000 samples). What I am finding is a relatively large variance in the loss function 2%-10% which makes it hard to identify which ...
2
votes
1answer
1k views

Which is better: Out of Bag (OOB) or Cross-Validation (CV) error estimates?

I have seen other posts in this forum but didn't find any convincing answer. Random Forest has an another way of tuning hyperparameter via OOB by design. OOB and CV are not the same as OOB error is ...
0
votes
1answer
91 views

Can the learning rate be considered both a parameter AND a hyper-parameter?

Here is my understanding of those 2 terms: Hyper-parameter: A variable that is set by a human before the training process starts. Examples are the number of hidden-layers in a Neural Network, the ...
1
vote
1answer
10k views

How to implement Python's MLPClassifier with gridsearchCV?

I am trying to implement Python's MLPClassifier with 10 fold cross-validation using gridsearchCV function. Here is a chunk of my code: ...
1
vote
1answer
560 views

Bounds for MLP hyperparameter search

I'm trying to optimize a neural network architecture for a particular problem, but there just seems to be so many hyperparameters that I'm concerned that there are much better options that I'm not ...
2
votes
1answer
2k views

How can you decide the window size on a pooling layer?

On the convolutional neural network, there used one or more pooling layers. As far as I know many tutorials instruct you to set it either 2 or ...
1
vote
0answers
96 views

Experimental design - hyperparameter optimization

I'm reading a paper by Bergstra and Bengio (2012) on random search for hyperparameter optimization. I'm confused by their graph and explanation in Section 2.2: "Figure 2 illustrates the results of a ...
2
votes
1answer
534 views

h2o, different stopping metric leads to different optimal for hyperparameters

I want to choose the "optimal" hyperparameters for gbm. So I run the following code using the h2o package ...
3
votes
1answer
2k views

Efficient way to optimise hyper parameter for network with multiple inputs?

I am currently looking for a way in which a network with multiple inputs can optimise its hyper parameter... scikit-learn has gridsearch CV but Keras only supports single inputs using the scikit-...
0
votes
1answer
472 views

Is GridSearchCV computing SVC with rbf kernel and different degrees?

I'm running a GridSearchCV with a OneVsRestClasssifer using SVC as an estimator. This is the ...
0
votes
1answer
274 views

Can't reproduce results from GridSearchCV?

I am trying to find optimized n_neighbors value for KnearestClassifier using GridSearchCV. I am able to get optimized parameters but when I enter those in my classifier results don't match with ...
1
vote
0answers
143 views

How do I choose the optimal parameters for reliefF

For feature selection I use reliefF provided by matlab. The reliefF function offers a parameter k to influence its ouput, additionaly for my specific task I can also vary a window length l on which ...
0
votes
1answer
2k views

XGboost classification with very small data set

I have a general question regarding XGboost and especially the n_rounds parameter, regarding small datasets. Normally I tune the n_rounds parameters by cross-validation, but what if you have too less ...