Questions tagged [hyperparameter]

Hyperparameters of a model are the kind of parameters that cannot be directly learned during training but are set beforehand. Hyperparameters can define, for example, the complexity of the model or its capacity to learn.

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

Track validation_curve during hyperparameter optimization

To study the influence of a single (hyper-)parameter, I use validation_curve: ...
1
vote
2answers
30 views

is learning rate schedule a hyperparameter? [duplicate]

I believe term ‘learning rate schedule’ is a certain solution for tuning the learning rate. But at the same time, every parameter evaluating the parameter itself can be called a hyperparameter. So can ...
3
votes
1answer
78 views

Grid search or gradient descent?

Assume we have a neural network and one if its activation functions is a function of parameter a. We want to find the weights and parameter a that leads to the minimum loss on the validation set which ...
0
votes
1answer
33 views

Difference between validation and prediction

As a follow-up to Validate via predict() or via fit()? I wonder about the difference between validation and prediction. To keep it simple, I will refer to train, <...
1
vote
1answer
50 views

Hyperparameter tuning and cross validation

I have some confusion about proper usage of cross-validation to tune hyperparameters and evaluate estimator performance and generalizeability. As I understand it, this would be the process you ...
0
votes
1answer
26 views

Similarity of XGBoost models?

Is xgboost with n_estimators = 100 and learning_rate = 0.1, same as xgboost with n_estimators = 50 and learning_rate = 0.2 ?
1
vote
1answer
27 views

Accuracy noise patterns during model training

I'm training a logistic regression model on a small dataset. I have about 1300 samples that I split into a training and a testing set (70% and 30% respectively). The training seems ok, however when I ...
0
votes
1answer
68 views

How to choose the model parameters (RandomizedSearchCV, .GridSearchCV) or manually

Faced with the task of selecting parameters for the lightgbm model, the question accordingly arises, what is the best way to select them? I used the RandomizedSearchCV method, within 10 hours the ...
1
vote
0answers
19 views

Analyzing the search space of hyperparameter optimization

My goal is to train a CNN via transfer learning on a given dataset and to analyze and document the training process. I selected a few CNN architectures and hyperparameters to perform a random search. ...
1
vote
0answers
8 views

Cross-validation for Timeseries Counterfactual Analysis

We are looking to predict counterfactual states from time-series data. In our problem we are looking to determine the energy savings from a grid-installed device that is varied on and off for many ...
0
votes
1answer
41 views

How to optimize hyperparameters in stacked model?

I was wondering whether somebody could explain how to optimize hyperparameters for the base learners and meta algorithm when stacking? In many tutorials they seem to be plucked out of thin air! ...
0
votes
0answers
25 views

Hyperopt Model runs with 0 seconds duration

I use Hyperopt for Random Forest Regression hyperparameter tuning. my parameterspace is : ...
1
vote
2answers
32 views

Parallel hyperparameter optimization techniques?

Most hyperparameter optimization technique want to evaluate points one by one. I have an expensive optimization problem, but i can run hundreds of evaluations in parallel. The dimension of the problem ...
1
vote
1answer
84 views

KNN RandomizedSearchCV typerror

While trying to study a binary classification problem with KNN and trying to tune the parameters of the model I'm getting a typerror that I quite don't understand. Is a parameter missing or something? ...
2
votes
0answers
140 views

Error while trying to do hyperparameter tuning using hyperas

I am getting a syntax error while using hyperas and am not sure why. My code: ...
1
vote
0answers
16 views

Slowly decreasing validation / training cost and their abnormal values

I have a dataset of size ~100,000 of images, I'm training a CNN model on them for regression. optimizer: Adam batch_size: 64 Number of epochs: 50 When I set the ...
0
votes
0answers
33 views

Confused about hyper-parameter search and network architecture search in NAS

I want to understand automatic Neural Architecture Search (NAS). I read already multiple papers, but I cannot figure out what the actual search space of NAS is / how are classical hyper-parameters ...
1
vote
1answer
15 views

A doubt about the GridSearchCV function in Sklearn?

When creating different hyperparameter combinations, does the function evaluate combination 1 on the same fold as combination 2? As in, are the folds the same across combinations? I understand that ...
1
vote
2answers
73 views

Is it a good idea to tune the number of folds for cross validation when tuning hyperparameters of RF

I'm new to data science. I'm trying to get the best model for Random Forest. Unfortunately, I'm not sure if my idea can produce a good generalized model. 1) I have split data to TrainingSet (70%) and ...
0
votes
0answers
47 views

Has anybody used alternative hyperparameter optimization techniques (other than default one) in SK-Learn?

I've been using Sklearn for Gaussian process regression that has L-BFGS-B (“fmin_l_bfgs_b”) as a default optimization algorithm. I want to implement some other ...
3
votes
1answer
101 views

Hyperopt vs Default Values

When I use the hyperopt library to tune my Random Forest classifier, I get the following results: Hyperopt estimated optimum {'max_depth': 10.0, 'n_estimators': 300.0} However, when I train the ...
0
votes
1answer
34 views

Generative network understanding

I was going through GAN's notebook by fchallot on Generative Adversarial Networks where, in the Generator Network, he creates a Dense layer with $16*16 * 128$ (...
2
votes
0answers
109 views

How to handle the parameter space of neural networks?

This question is very broad (and might even be closed as "too broad"). It can be considered as a beginners question, because it is largely about getting started in terms of heading into a direction ...
1
vote
0answers
446 views

Bayesian optimization for a Light GBM Model

I am able to successfully improve the performance of my XGBoost model through Bayesian optimization, but the best I can achieve through Bayesian optimization when using Light GBM (my preferred choice) ...
0
votes
0answers
38 views

Reasonable hyperparameters for NuSVR?

I'm looking for reasonable hyperparameters grid for NuSVR. For now I have: ...
1
vote
1answer
83 views

Tuning C hyper parameter in Soft Margin SVM in Matlab

How to tune the C 'BoxConstraint' hyperparameter in soft margin SVM to get the best optimal value?
1
vote
0answers
147 views

SVM hard and soft margins in matlab,

I am comparing the performances of several SVM models in matlab using the fitcsvm function, and I want to double check that I am using the correct syntax for hard ...
0
votes
0answers
147 views

SVM hyperparameters using Matlab's fitcsvm and OptimizeHyperparameters

I am building SVM models and will compare their performances, linear vs RBF, and I'm using OptimizeHyperparameters to get best hyperparameters C (BoxConstraints) ...
7
votes
1answer
205 views

Is it OK to try to find the best PCA k parameter as we do with other hyperparameters?

Principal Component Analysis (PCA) is used to reduce n-dimensional data to k-dimensional data to speed things up in machine learning. After PCA is applied, one can check how much of the variance of ...
2
votes
2answers
261 views

Setting best SVM hyper parameters

I have a non linear data set, and I am using SVM (RBF kernel) to build a classification model, but not sure how to set the best hyperparameters of the SVM, C and gamma in Matlab ...
0
votes
1answer
108 views

What is the possible range of SVR parameters range?

I'm working on a regression problem. While tunning the Parameters of SVR I got the following values c=100, gamma= 10 and epsilon =100. For which I got 95 percent r-square. My question is what is the ...
1
vote
0answers
48 views

Problem about tuning hyper-parametres

I have tried GridSearchCV and BayesSearchCV for tuning my LightGBM algorithm (for binary classification). I have used 10 iterations and I have indicated scoring ="roc_auc" In the first iteration, I ...
9
votes
2answers
609 views

What is the most efficient method for hyperparameter optimization in scikit-learn?

An overview of the hyperparameter optimization process in scikit-learn is here. Exhaustive grid search will find the optimal set of hyperparameters for a model. The downside is that exhaustive grid ...
3
votes
2answers
83 views

How to set hyperparameters in SVM classification

I am studying image classification using SVMs and it is generally defined as so... N = number of training examples W = is the weights f(x, W) = dot product λ is explained to be set through cross-...
4
votes
1answer
102 views

xgboost or lightgbm to handle Binomial problems

I have a dataset containing a column of trials, a column of successes and other features; and, obviously, I can generate a probability column. I would like to use gradient boosting methods (like ...
1
vote
0answers
20 views

Any heuristic for minimal DCGAN latent space dimension?

I am highly interested in approaching minimal latent space dimension (as many other may be) for DCGANs or autoencoders. In this example of DCGAN on the MNIST dataset, the person uses a 100-...
0
votes
0answers
38 views

How to adjust deep learning parameters using Particle swarm optimization (PSO)?

As success of deep learning depends upon appropriately setting of its parameters to achieve high-quality results. The number of hidden layers and the number of neurons in each layer of a deep machine ...
0
votes
1answer
37 views

hypeparameters tuning neural network according to loss vs according to scoring function

During hyperparameters tuning we select a metric to measure performance of the model. Example of metrics : f1 score, precision, recall, AUC ... In general, for the training of neural networks, back-...
0
votes
0answers
15 views

Hyper-parameter tuning when you don't have an access to the test data

I'm building models for SQUAD (Stanford Question Answering) dataset (https://rajpurkar.github.io/SQuAD-explorer). Stanford doesn't release its test set. It only provides us with training and dev ...
6
votes
4answers
938 views

Which is first ? Tuning the parameters or selecting the model

I've been reading about how we split our data into 3 parts; generally, we use the validation set to help us tune the parameters and the test set to have an unbiased estimate on how well does our model ...
3
votes
1answer
548 views

Hyperparameter tuning for stacked models

I'm reading the following kaggle post for learning how to incorporate model stacking http://blog.kaggle.com/2016/12/27/a-kagglers-guide-to-model-stacking-in-practice/ in ML models. The structure ...
2
votes
0answers
203 views

How to tune parameters for Time Series Analysis, when forecasting is only dominated by one feature and error is not getting reduced?

I am trying to predict time series based on 150 features. When I plot correlation of these features, I am getting 20 features with more or less importance but every model I use, it is completely ...
1
vote
0answers
13 views

How to optimize the separation of two distributions from binary classfication

Given a sample where for each individual a classification is predetermined (e.g. sick or not) and 5 random variables are measured. The random variables are on the same scale but from differnt bins. E....
1
vote
0answers
68 views

std::bad_alloc with densenet and hyperas

I am filing this issue after being stagnated here for couple of weeks. I am using hyperas to find the hyperparameters for my network, Densenet. My issue here is that my evaluation always fails with ...
0
votes
1answer
65 views

Denoising Autoencoder Parameter Search

I have ran a hyperparameter search for a denoising autoencoder and the results suggest I should make the sizes of my hidden layers as large as possible (within the range of values I allowed it to ...
1
vote
1answer
103 views

What is difference between final episodes of training and test in DQN?

What is difference between running in final episode of training mode and running in test mode in DQN? Is there any difference more than after training and tune the hyper-parameters, we test for one ...
4
votes
1answer
83 views

How to make it possible for a neural network to tune its own hyper parameters?

I am curious about what would happen to hyperparameters when they would be set by a neural network itself or by creating a neural network that encapsulates and influences the hyperparameters of the ...
1
vote
0answers
206 views

How to perform platt scaling for hyperparameter-optimized model?

I'm using Python and have a best estimator from a grid search. Wanted to be able to calibrate the probability output accordingly, but would like to know more about implementing platt scaling. From ...
3
votes
2answers
2k views

How to choose the random seed?

I understand this question can be strang, but how do I pick the final random_seed for my classifier? Below is an example code. It uses the ...
2
votes
1answer
161 views

Benefits of using Deep Learning-specific hyperparameter optimization tools vs. sklearn?

There are quite a few library for hyperparameter optimization that are specific to Keras or other Deep Learning libraries, like Hyperas or Talos. My question is, what's the main benefit of using ...