Questions tagged [hyperparameter]

Hyperparameters of a model are the kind of parameters that cannot be directly learned during training but are set beforehand. Hyperparameters can define, for example, the complexity of the model or its capacity to learn.

Filter by
Sorted by
Tagged with
3
votes
1answer
1k views

Automated tuning of Hyperparameter

Are there any advanced packages that allows automated tuning of hyperparameters for neural network and traditional machine learning algorithms like XGBoost, random forest (using method like Bayesian, ...
1
vote
2answers
504 views

How to tune bandwidth in machine learning kernel model?

Gaussian kernel $k(x,y) = \exp(-\lVert x-y \rVert^2/\sigma^2)$ has a hyperparameter $\sigma$. I know grid search cross validation, but this would require a lot of computation since computational ...
2
votes
4answers
1k views

Is it better to optimize hyperparameters or run multiple epochs?

Whenever I train a neural network I only have it go through a few epochs ( 1 to 3). This is because I am training them on a bad CPU and it would take some time to have the neural network go though ...
3
votes
1answer
506 views

Hyper parameters and ValidationSet

Please correct me if I am wrong. "Training Set is used for calculating parameters of a machine learning model, Validation data is used for calculating hyperparameters of the same model (we use same ...
1
vote
1answer
941 views

Is there any alternative to L-BFGS-B algorithm for hyperparameter optimization in Scikit learn?

The Gaussian process regression can be computed in scikit learn using an object of class GaussianProcessRegressor as: ...
1
vote
1answer
3k views

Optimising Kernel parameters using training data in GaussianProcessRegressor of Scikit-learn

I want to optimize the Kernel parameters or hyper-parameters using my training data in GaussianProcessRegressor of Scikit-learn.Following is my query: My training datasets are: X: 2-D Cartesian ...
3
votes
2answers
90 views

Comparison of machine learning approaches for a topic in a scientific paper

As part of my master's thesis, I have made a prediction of data with approaches of machine learning in a topic where are no papers yet. The topic is a regression problem for which several machine ...
4
votes
1answer
467 views

Why do we need the hyperparameters beta and alpha in LDA?

I'm trying to understand the technical part of Latent Dirichlet Allocation (LDA), but I have a few questions on my mind: First: Why do we need to add alpha and gamma every time we sample the equation ...
2
votes
1answer
229 views

Do we need to increase training data size when increasing dropouts?

I am using a fully connected feed forward neural network built using keras for text classification. It consists of 3 hidden layer. I am planning to add a dropout layer after each hidden layer to ...
2
votes
1answer
204 views

Parameter tuning for machine learning algorithms

When it comes to the topic of tuning parameters, most of the time you read grid search. But if you have 6 parameters, for which you want to test 10 variants, you get to 10^6 = 1000000 runs. Which in ...
3
votes
1answer
608 views

Basic method of optimizing hyperparameters

I recently read the LIPO blog post on the dlib blog: http://blog.dlib.net/2017/12/a-global-optimization-algorithm-worth.html It mentions that it can be used for optimizing hyperparameters of eg ...
2
votes
1answer
62 views

How to think about prediction error that is not convex in hyperparameter, or over the course of training

Take the following case of a hyperparameter and prediction error: Imagine that the hyperparameter is a L2 penalty or a dropout rate -- something that we think that should have a single sweet spot -- ...
2
votes
1answer
260 views

Strategies for automatically tuning the hyper-parameters of deep learning models

I'm interested in trying to develop a framework for automatically tuning the hyperparameters of deep learning models and I'm looking for advice / references to resources that may be useful and ...
1
vote
1answer
1k views

Bounds for MLP hyperparameter search

I'm trying to optimize a neural network architecture for a particular problem, but there just seems to be so many hyperparameters that I'm concerned that there are much better options that I'm not ...
1
vote
3answers
288 views

Grid Search and High Variance

I am currently trying to optimise some parameters on my model (15000 samples). What I am finding is a relatively large variance in the loss function 2%-10% which makes it hard to identify which ...
4
votes
1answer
4k views

Which is better: Out of Bag (OOB) or Cross-Validation (CV) error estimates?

I have seen other posts in this forum but didn't find any convincing answer. Random Forest has an another way of tuning hyperparameter via OOB by design. OOB and CV are not the same as OOB error is ...
0
votes
1answer
366 views

Can the learning rate be considered both a parameter AND a hyper-parameter?

Here is my understanding of those 2 terms: Hyper-parameter: A variable that is set by a human before the training process starts. Examples are the number of hidden-layers in a Neural Network, the ...
6
votes
1answer
6k views

How can you decide the window size on a pooling layer?

On the convolutional neural network, there used one or more pooling layers. As far as I know many tutorials instruct you to set it either 2 or ...
1
vote
0answers
199 views

Experimental design - hyperparameter optimization

I'm reading a paper by Bergstra and Bengio (2012) on random search for hyperparameter optimization. I'm confused by their graph and explanation in Section 2.2: "Figure 2 illustrates the results of a ...
2
votes
1answer
919 views

h2o, different stopping metric leads to different optimal for hyperparameters

I want to choose the "optimal" hyperparameters for gbm. So I run the following code using the h2o package ...
3
votes
1answer
2k views

Efficient way to optimise hyper parameter for network with multiple inputs?

I am currently looking for a way in which a network with multiple inputs can optimise its hyper parameter... scikit-learn has gridsearch CV but Keras only supports single inputs using the scikit-...
1
vote
1answer
2k views

Is GridSearchCV computing SVC with rbf kernel and different degrees?

I'm running a GridSearchCV with a OneVsRestClasssifer using SVC as an estimator. This is the ...
0
votes
1answer
2k views

Can't reproduce results from GridSearchCV?

I am trying to find optimized n_neighbors value for KnearestClassifier using GridSearchCV. I am able to get optimized parameters but when I enter those in my classifier results don't match with ...
1
vote
0answers
238 views

How do I choose the optimal parameters for reliefF

For feature selection I use reliefF provided by matlab. The reliefF function offers a parameter k to influence its ouput, additionaly for my specific task I can also vary a window length l on which ...
1
vote
1answer
5k views

XGboost classification with very small data set

I have a general question regarding XGboost and especially the n_rounds parameter, regarding small datasets. Normally I tune the n_rounds parameters by cross-validation, but what if you have too less ...
1
vote
0answers
181 views

Global vs. local bias-variance tradeoff

In the standard example of decomposing the MSE into Bias, Variance and Irreducible error: $$MSE(x) = \left(\mathbb{E}[\hat{f}(x)] - f(x) \right)^2 + \mathbb{E}\left[\left(\hat{f}(x) - f(x)\right)^2\...
1
vote
0answers
3k views

Why is my loss so high?

I am struggling to understand why i am getting such a high loss/val_loss rate on my training. I am training a regression network. I've normalized the input data to range between -1 to 1, and left the ...
6
votes
1answer
990 views

Neural Network Golf: smallest network for a certain level of performance

I am interested in any data, publications, etc about what is the smallest neural network that can achieve a certain level of classification performance. By small I mean few parameters, not few ...
3
votes
1answer
2k views

How can the process of hypertuning of XGBoost parameters be automated?

I'm using xgboost for training a model on a data with extreme class imbalance. After referring from here. After performing grid search and some manual settings, I ...
6
votes
1answer
4k views

Overfitting for minority class after SMOTE w/ random forests

I used SMOTE to make a predictive model, with class 1 having 1800 samples and 35000+ of class 0 samples. Hence, as per SMOTE, synthetic samples were created and the random forest was trained. However,...
2
votes
2answers
2k views

Tune hyperparameters for cost-sensitive classification

I have an unbalanced data set with about 8% of negative examples. The goal is to minimize false negatives given a cost matrix. It seems like SVM (with radial kernel) and random forest work best. How ...

1 2
3