Questions tagged [hyperparameter]

Hyperparameters of a model are the kind of parameters that cannot be directly learned during training but are set beforehand. Hyperparameters can define, for example, the complexity of the model or its capacity to learn.

Filter by
Sorted by
Tagged with
3
votes
1answer
199 views

Benefits of using Deep Learning-specific hyperparameter optimization tools vs. sklearn?

There are quite a few library for hyperparameter optimization that are specific to Keras or other Deep Learning libraries, like Hyperas or Talos. My question is, what's the main benefit of using ...
0
votes
1answer
15 views

Can we make an AI to fine tune other AI hyper parameters?

Every time AI gurus talk about fine tuning hyper parameters, they more or less say it's trial and error. But can't we make an AI to tell AI what its hyper parameters should be?
0
votes
2answers
30 views

select hyperparameters using Latin hypercube sampling (LHS) from a large matrix/grid of parameter combinations

I have a matrix with each row corresponds to a hyperparameter for the XGBoost model. There are seven parameters to tune in XGBoost (as shown below: nrounds/iterations, max_depth, eta, gamma, ...
39
votes
6answers
36k views

What is the difference between model hyperparameters and model parameters?

I have noticed that such terms as model hyperparameter and model parameter have been used interchangeably on the web without prior clarification. I think this is incorrect and needs explanation. ...
-2
votes
1answer
28 views

Help with Classification using scikit-learn models [closed]

I'm using the Titanic data set to classify the missing Cabins. There is a lot of missing Cabin values. My objective is just to assign the letter of the Cabin without the room number. So, I'm just ...
1
vote
1answer
21 views

What does updated alpha mean in LDA model?

I'm trying to understand LDA model by reading through implementations of the algorithm. Many implementations update alpha during training iterations with codes like: ...
2
votes
2answers
9k views

Hyper-parameter tuning of NaiveBayes Classier

I'm fairly new to machine learning and I'm aware of the concept of hyper-parameters tuning of classifiers, and I've come across a couple of examples of this technique. However, I'm trying to use ...
2
votes
2answers
1k views

How can I tune LSTM hyperparameters?

If anyone is there to answer these, that'll be great. I'm in the midst of a Final Year Project on LSTM. Currently, I’m stuck and confused over LSTM codes. There are 4 hyperparameters that I can play ...
1
vote
1answer
31 views

Two questions on hyper-parameter tuning

Question 1: In the example of logistic regression, I often see the regularization constant and penalty methods being tuned by a grid search. However, it seems like there are a lot more options for ...
0
votes
1answer
47 views

Generative network understanding

I was going through GAN's notebook by fchallot on Generative Adversarial Networks where, in the Generator Network, he creates a Dense layer with $16*16 * 128$ (...
1
vote
0answers
11 views

Can I dynamically change the hyper-parameters of a model?

Question Can I apply different hyper-parameters for different training sets? I can see the point of using the shared parameters but I cannot see the point of using shared hyper-parameters. The ...
0
votes
2answers
548 views

Random Forest - Explanation Parameter

I got some question about the "standard" parameter from a random forest. Following I write my understanding about these parameters. I would be glad if I could confirm my understanding or correct it. :)...
1
vote
0answers
46 views

XGBoost: Typical gamma and min_child_weight range

What is the typical accepted range of gamma and min_child_weight parameters for the XGBoost algorithm? Is the range of min_child_weight correlated with the number of feature or samples in the training ...
1
vote
0answers
18 views

How to compare hyperparameter tuning in R and Python

I tried random forest in both R (Caret) and Python (Scikit-learn), but the results differ drastically. Pearson correlation between predicted value and actual value was 0.2 in python whereas 0.8 in R. ...
0
votes
2answers
33 views

Using Transaction Amount to Guide Learning in an Fraud Detection Machine Learning Model

I am currently using transaction amount as a feature in an XGBoost classification model designed to identify fraudulent transactions. Furthermore, transaction amount is bounded for this problem ...
1
vote
2answers
72 views

Parallel hyperparameter optimization techniques?

Most hyperparameter optimization technique want to evaluate points one by one. I have an expensive optimization problem, but i can run hundreds of evaluations in parallel. The dimension of the problem ...
1
vote
0answers
63 views

SVM C vs gamma hyperparameter tuning

While running SVC(), how we can hyperparameter tune C vs gamma combination? I could see changes in C and gamma are impacting the accuracy differently. Also, what I understand about C and gamma are: C ...
0
votes
1answer
238 views

BERT minimal batch size

Is there a minimum batch size for training/re-fining a BERT model on custom data? Could you name any cases where a mini batch size between 1-8 would make sense? Would a batch size of 1 make sense at ...
4
votes
1answer
24k views

How to implement Python's MLPClassifier with gridsearchCV?

I am trying to implement Python's MLPClassifier with 10 fold cross-validation using gridsearchCV function. Here is a chunk of my code: ...
0
votes
0answers
16 views

How to tune the hyperparameters in graphical models such as a Markov random field?

What's a good way to tune the hyperparemter of a Markov random field (MRF) with a Gaussian prior that's not uncommon in image segmentation tasks? The structure of the graphical model is not a big ...
0
votes
0answers
73 views

Hyper tuning reduce the accuracy score, why?

I have performed hyper tuning grid CV search on KNN model. The actual accuracy score for my KNN was accuracy of 42.31 % without performing hyper tuning. However, after performing hyper tuning, the ...
0
votes
1answer
39 views

Hyperparameter tuning of neural networks using Bayesian Optimization

One of the assumptions for finding good hyperparameters using Bayesian optimization (GP) is that the unknown function is smooth. Is this assumption valid for neural networks or at least for most of ...
2
votes
2answers
70 views

Faster grid search with small dataset to derive best params instead of full dataset?

I have a dataset of 300 000 rows and an ensemble model, which include grid search to find the best params of every algorithm. Unfortunately the grid search needs to long and I have problems to ...
35
votes
6answers
7k views

How to set the number of neurons and layers in neural networks

I am a beginner to neural networks and have had trouble grasping two concepts: How does one decide the number of middle layers a given neural network have? 1 vs. 10 or whatever. How does one decide ...
0
votes
1answer
22 views

How to distinguish between different values of a hyperparameter in communication?

From Wikipedia: In machine learning, a hyperparameter is a parameter whose value is used to control the learning process. If we go by the definition of parameter in What's the difference between an ...
1
vote
0answers
68 views

Adaptive Resampling in Caret with Pre-specified Validation Set

I was wondering if this is the correct way to get adaptive sampling in caret working with a pre-specified validation set using index. I can get this to work using the 'cv' method in caret like so <...
0
votes
0answers
89 views

Good mathematically explained algorithm for Hyperparameter Optimization (Bayesian) for implementing in Java

I have implemented Random Forest, Bagging, Gradient Boosting, etc... in java myself. It took a long time to complete these machine learning algorithm codes. But at last all of them are well running. ...
2
votes
1answer
300 views

Hyperparameters in Gaussian Process

My academical background is in physics and analysis (PDE's), but now I'am reading about data science. I'm reading about Gaussian Process implementation in Sci-Kit Learn I could not find a simple ...
19
votes
4answers
20k views

Hyperparameter search for LSTM-RNN using Keras (Python)

From Keras RNN Tutorial: "RNNs are tricky. Choice of batch size is important, choice of loss and optimizer is critical, etc. Some configurations won't converge." So this is more a general question ...
0
votes
1answer
71 views

DBSCAN Clustering

I used K-means to get the number of clusters for my data(Elbow Method). Then I was trying to see if for some specific hyperparameters can we get the same number of clusters for DBSCAN. I tried Brute-...
4
votes
1answer
279 views

How many epochs to run during hyperparameter search?

If I'm doing a hyperparameter search and comparing two different hyperparameters (but not number of epochs), is there some established rule of thumb for how many epochs to run? If I just compare ...
0
votes
1answer
104 views

Final Model fitting - subset vs entire training data

If I used a subset of the entire available training data for model tuning and hyperparamater selection, should I fit the final model to the subset training dataset or the entire available training ...
3
votes
1answer
413 views

Hyperparameter tuning and cross validation

I have some confusion about proper usage of cross-validation to tune hyperparameters and evaluate estimator performance and generalizeability. As I understand it, this would be the process you would ...
4
votes
2answers
330 views

xgboost or lightgbm to handle Binomial problems [duplicate]

I have a dataset containing a column of trials, a column of successes and other features; and, obviously, I can generate a probability column. I would like to use gradient boosting methods (like ...
1
vote
1answer
303 views

XGboost and regularization

Does the XGBClassifier method utilizes the two regularization terms reg_alpha and reg_lambda,...
0
votes
0answers
107 views

How to grid search class.weights hyper parameter in Ranger?

I am currently using ranger for binary classification. My dataset is highly imbalanced (10:1). I went over the documentation, and it appeared to me that ...
4
votes
2answers
4k views

How to choose the random seed?

I understand this question can be strange, but how do I pick the final random_seed for my classifier? Below is an example code. It uses the ...
1
vote
1answer
30 views

Tuning SVM C parameter

I would like to ask for help regarding my model. I have a dataset of preprocessed images and I performed a binary classification with SVM on Python. I tuned the value of the c parameter from 0.001 to ...
-1
votes
1answer
184 views

Denoising Autoencoder Parameter Search

I have ran a hyperparameter search for a denoising autoencoder and the results suggest I should make the sizes of my hidden layers as large as possible (within the range of values I allowed it to ...
0
votes
1answer
100 views

Hill Climbing Algorithm - Optimum Step Size

I am implementing a standard hill climbing algorithm to optimise hyper-parameters for a predictive model. The hill climbing algorithm is being applied as part of a two-stage approach: Apply grid ...
3
votes
2answers
778 views

XGBoost and Random Forest: ntrees vs. number of boosting rounds vs. n_estimators

So I understand the main difference between Random Forests and GB Methods. Random Forests grow parallel trees and GB Methods grow one tree for each iteration. However, I am confused on the vocab used ...
1
vote
2answers
167 views

Is the search space of Hyperparameters Continuous or Discrete?

I am looking into hyper-parameter tunning and was curious about whether the search space is considered continuous or discrete? My understanding of both those cases: 1. Continuous would make it 'easier'...
1
vote
1answer
27 views

Asynchronous Hyperparameter Optimization - Dependency between iterations

When using Asynchronous Hyperparameter Optimization packages such as scikit optimize or hyperopt with cross validation (e.g., cv = 2 or 4) and setting the number of iteration to N (e.g., N=100), ...
1
vote
0answers
24 views

Classification Model showing different accuracy for SAME data?

This is my first post here, so kindly pardon any commonplace errors. So, i have been training an XGBoost multi-class model on Google Colab. I am using a balanced dataset, with 31000 rows, where each ...
3
votes
2answers
508 views

Shuffle the data before splitting into folds

I am running a 4-folds cross validation hyperparameter tuning using sklearn's 'cross_validate' and 'KFold' functions. Assuming that my training dataset is already shuffled, then should I for each ...
1
vote
0answers
18 views

Techniques for hyperparameter search in non-stationary environments

I'm tuning a supervised machine learning model over time, also called incremental learning. I do not want to assume the environment is non-stationary. Grid search and random search do not appear to ...
3
votes
1answer
785 views

Small number of estimators in gradient boosting

I am tuning a regression gradient boosting-based model to determine the appropriate hyperparameters using 4-folds cross validation. More specifically, I am using XGBoost and lightGBM for the models ...
4
votes
1answer
227 views

Hyperopt vs Default Values

When I use the hyperopt library to tune my Random Forest classifier, I get the following results: Hyperopt estimated optimum {'max_depth': 10.0, 'n_estimators': 300.0} However, when I train the ...
2
votes
1answer
272 views

Hyperparameter optimization performance comparison

I have used Bayesian optimization for hyperparameter tuning in a machine learning model. What is the best way to compare the performance of network with and without Bayesian optimization? I found some ...