Questions tagged [hyperparameter]

Hyperparameters of a model are the kind of parameters that cannot be directly learned during training but are set beforehand. Hyperparameters can define, for example, the complexity of the model or its capacity to learn.

Filter by
Sorted by
Tagged with
0 votes
0 answers
19 views

Relation between batch size, number of steps, and learning rate

Taking alphazero training setup as a reference: 700k total steps batch-size of 4096 initial LR of 0.2 What would be an equivalent setup for a batch-size of 1024? ...
user avatar
  • 101
1 vote
1 answer
21 views

Different results between hyperparameter optimisation and actual training/val values

If I want to do a hyperparameter optimisation on a dataset using e.g. hyperband or random search, I note that some of the models being randomly chosen seem to have rather good R2 scores, MSE etc. I ...
user avatar
  • 61
1 vote
0 answers
12 views

Feature Selection - Comparing Performance of different size datasets

If I have training data X, with N features, and I do feature selection, and discover n of <...
user avatar
  • 61
0 votes
0 answers
5 views

Is this XGBoost model tending to overfit?

Here is the list of hyperparameters that I used: ...
user avatar
0 votes
0 answers
29 views

The accuracy depends on the hyper-parameter in a strongly non-monotinic way

I have a data set labelled with a binary classes. I calculated the principal components from the data, then made the PC transformation. The goal is to find an optimal number of PCs so that the binary ...
user avatar
0 votes
1 answer
24 views

Why does hyperparameter tuning occur on validation dataset and not at the very beginning?

Despite doing/using it a few times, I'm still slightly confused by the use of a validation set for hyper parameter tuning. As far as I can tell, I choose a model, train it on training data, assess ...
user avatar
  • 61
0 votes
1 answer
171 views

How to suppress "Estimator fit failed. The score on this train-test" warning message?

I am working on hyper-tuning random forest classifier with following parameters in random search CV ...
user avatar
0 votes
0 answers
8 views

How to choose max layers and units to search over in hyper parameter tuning

When performing any hyper parameter tuning, let's say random search for simplicity, and I want to search over a minimum to max units/nodes in a layer, and a minimum to max number of layers, are there ...
user avatar
  • 61
0 votes
0 answers
9 views

Efficient Searching for a basis of information as a hyperparameter in a large possible hyperparameter space

I have a set of inputs, let's call them 'I', that can be fed through a complicated group of functions to produce/calculate a wide variety of outputs (let's call them 'O'). I want to find a subset of ...
user avatar
  • 35
0 votes
0 answers
13 views

Rules, rules of thumb, intuitions, on how to set up the best possible hyperparameter search

When I set up my neural networks, I really have very little idea what I'm doing in advance. It may just be a bit of educated guesswork as to "it may need a few layers only" or "this ...
user avatar
  • 61
2 votes
1 answer
54 views

Estimating Length of Hyperband Trials in Advance

I would like to use the (Keras/Tensorflow) hyperband tuning algorithm more than the Keras random search, for instance, when testing hyperparameters. With random search I can set max trials and get a ...
user avatar
  • 61
0 votes
0 answers
12 views

Activation Function Hyperparameter Optimisation

If I have a model, say: ...
user avatar
  • 61
0 votes
0 answers
29 views

Drop Out in Hyperparameter Optimisation

Is it correct to add dropout to each layer and that it is done as in the below example? class MyHyperModel(kt.HyperModel): def build_model(self, hp): ...
user avatar
  • 61
0 votes
0 answers
14 views

Should hyperparameter optimisation focus on many trials (models) lower epochs first, then a second round with few models, many epochs?

Rather than a hyperparameter optimisation with kt.tuners.RandomSearch, say, that does (option A), say X model trials (e.g. 100), Y epochs each (say 100, so a total of 10,000 epochs across all models) ...
user avatar
  • 61
0 votes
1 answer
24 views

Training Loss or Validation Loss for Hyperparameter Optimisation

When performing HO, should I be looking to train each model (each with different hyperparameter values, e.g. with RandomSearch picking those values) on the training data, and then the best one is ...
user avatar
  • 61
0 votes
0 answers
19 views

Scikit Learn Random Forest Classifier Hyperparameter Min Target Sample Size

From reading the docs on Scikit Learn, I haven't been able to find an answer, but does anyone know if there is a way to specify to always include a specific number out of the max sample size of ...
user avatar
0 votes
0 answers
11 views

How to specify Search Space in Auto-Sklearn

I know how to specify Feature Selection methods and the list of the Algorithms used in Auto-Sklearn 2.0 ...
user avatar
  • 497
0 votes
0 answers
102 views

Xgboost taking some time to run vs hyperopt

Sorry for long post,im triying to run a xgb model but for some reason takes like 20 to 30 min(per run) with a specific set of hyperparams, but when i run hyperopt to get best params, takes like 7 ...
user avatar
0 votes
1 answer
685 views

Optimal batch size and number of epoch for BERT

I use this tutorial https://www.tensorflow.org/text/tutorials/classify_text_with_bert and get different accuracy depend on epoch numbers and batch sizes. What's optimal parameters?
user avatar
1 vote
1 answer
31 views

Hyperparameter searching when there is no development set

I have a train and a test set and no development (dev) set. I'm training a model on the train set and searching for the best hyperparameters that can eventually maximize the accuracy of the test set (...
user avatar
  • 133
0 votes
1 answer
23 views

Can we make an AI to fine tune other AI hyper parameters?

Every time AI gurus talk about fine tuning hyper parameters, they more or less say it's trial and error. But can't we make an AI to tell AI what its hyper parameters should be?
user avatar
  • 101
-2 votes
1 answer
32 views

Help with Classification using scikit-learn models [closed]

I'm using the Titanic data set to classify the missing Cabins. There is a lot of missing Cabin values. My objective is just to assign the letter of the Cabin without the room number. So, I'm just ...
user avatar
1 vote
1 answer
70 views

What does updated alpha mean in LDA model?

I'm trying to understand LDA model by reading through implementations of the algorithm. Many implementations update alpha during training iterations with codes like: ...
user avatar
1 vote
2 answers
166 views

select hyperparameters using Latin hypercube sampling (LHS) from a large matrix/grid of parameter combinations

I have a matrix with each row corresponds to a hyperparameter for the XGBoost model. There are seven parameters to tune in XGBoost (as shown below: nrounds/iterations, max_depth, eta, gamma, ...
user avatar
  • 123
1 vote
1 answer
27 views

Can I apply different hyper-parameters for different sliding time windows?

Question Can I apply different hyper-parameters for different training sets? I can see the point of using the shared parameters but I cannot see the point of using shared hyper-parameters. The ...
user avatar
1 vote
0 answers
36 views

How to compare hyperparameter tuning in R and Python

I tried random forest in both R (Caret) and Python (Scikit-learn), but the results differ drastically. Pearson correlation between predicted value and actual value was 0.2 in python whereas 0.8 in R. ...
user avatar
0 votes
3 answers
54 views

Using Transaction Amount to Guide Learning in an Fraud Detection Machine Learning Model

I am currently using transaction amount as a feature in an XGBoost classification model designed to identify fraudulent transactions. Furthermore, transaction amount is bounded for this problem ...
user avatar
  • 23
1 vote
0 answers
202 views

XGBoost: Typical gamma and min_child_weight range

What is the typical accepted range of gamma and min_child_weight parameters for the XGBoost algorithm? Is the range of min_child_weight correlated with the number of feature or samples in the training ...
user avatar
0 votes
1 answer
2k views

BERT minimal batch size

Is there a minimum batch size for training/re-fining a BERT model on custom data? Could you name any cases where a mini batch size between 1-8 would make sense? Would a batch size of 1 make sense at ...
user avatar
0 votes
0 answers
386 views

Hyper tuning reduce the accuracy score, why?

I have performed hyper tuning grid CV search on KNN model. The actual accuracy score for my KNN was accuracy of 42.31 % without performing hyper tuning. However, after performing hyper tuning, the ...
user avatar
0 votes
1 answer
24 views

How to distinguish between different values of a hyperparameter in communication?

From Wikipedia: In machine learning, a hyperparameter is a parameter whose value is used to control the learning process. If we go by the definition of parameter in What's the difference between an ...
user avatar
2 votes
2 answers
184 views

Faster grid search with small dataset to derive best params instead of full dataset?

I have a dataset of 300 000 rows and an ensemble model, which include grid search to find the best params of every algorithm. Unfortunately the grid search needs to long and I have problems to ...
user avatar
  • 329
1 vote
0 answers
132 views

Adaptive Resampling in Caret with Pre-specified Validation Set

I was wondering if this is the correct way to get adaptive sampling in caret working with a pre-specified validation set using index. I can get this to work using the 'cv' method in caret like so <...
user avatar
  • 183
0 votes
1 answer
153 views

DBSCAN Clustering

I used K-means to get the number of clusters for my data(Elbow Method). Then I was trying to see if for some specific hyperparameters can we get the same number of clusters for DBSCAN. I tried Brute-...
user avatar
0 votes
1 answer
289 views

Final Model fitting - subset vs entire training data

If I used a subset of the entire available training data for model tuning and hyperparamater selection, should I fit the final model to the subset training dataset or the entire available training ...
user avatar
1 vote
1 answer
2k views

XGboost and regularization

Does the XGBClassifier method utilizes the two regularization terms reg_alpha and reg_lambda,...
user avatar
1 vote
1 answer
62 views

Tuning SVM C parameter

I would like to ask for help regarding my model. I have a dataset of preprocessed images and I performed a binary classification with SVM on Python. I tuned the value of the c parameter from 0.001 to ...
user avatar
0 votes
1 answer
304 views

Hill Climbing Algorithm - Optimum Step Size

I am implementing a standard hill climbing algorithm to optimise hyper-parameters for a predictive model. The hill climbing algorithm is being applied as part of a two-stage approach: Apply grid ...
user avatar
2 votes
3 answers
16k views

Hyper-parameter tuning of NaiveBayes Classier

I'm fairly new to machine learning and I'm aware of the concept of hyper-parameters tuning of classifiers, and I've come across a couple of examples of this technique. However, I'm trying to use ...
user avatar
8 votes
2 answers
2k views

XGBoost and Random Forest: ntrees vs. number of boosting rounds vs. n_estimators

So I understand the main difference between Random Forests and GB Methods. Random Forests grow parallel trees and GB Methods grow one tree for each iteration. However, I am confused on the vocab used ...
user avatar
1 vote
1 answer
63 views

Two questions on hyper-parameter tuning

Question 1: In the example of logistic regression, I often see the regularization constant and penalty methods being tuned by a grid search. However, it seems like there are a lot more options for ...
user avatar
0 votes
1 answer
54 views

Hyperparameter tuning of neural networks using Bayesian Optimization

One of the assumptions for finding good hyperparameters using Bayesian optimization (GP) is that the unknown function is smooth. Is this assumption valid for neural networks or at least for most of ...
user avatar
1 vote
2 answers
324 views

Is the search space of Hyperparameters Continuous or Discrete?

I am looking into hyper-parameter tunning and was curious about whether the search space is considered continuous or discrete? My understanding of both those cases: 1. Continuous would make it 'easier'...
user avatar
1 vote
1 answer
32 views

Asynchronous Hyperparameter Optimization - Dependency between iterations

When using Asynchronous Hyperparameter Optimization packages such as scikit optimize or hyperopt with cross validation (e.g., cv = 2 or 4) and setting the number of iteration to N (e.g., N=100), ...
user avatar
1 vote
0 answers
40 views

Classification Model showing different accuracy for SAME data?

This is my first post here, so kindly pardon any commonplace errors. So, i have been training an XGBoost multi-class model on Google Colab. I am using a balanced dataset, with 31000 rows, where each ...
user avatar
3 votes
2 answers
865 views

Shuffle the data before splitting into folds

I am running a 4-folds cross validation hyperparameter tuning using sklearn's 'cross_validate' and 'KFold' functions. Assuming that my training dataset is already shuffled, then should I for each ...
user avatar
1 vote
0 answers
25 views

Techniques for hyperparameter search in non-stationary environments

I'm tuning a supervised machine learning model over time, also called incremental learning. I do not want to assume the environment is non-stationary. Grid search and random search do not appear to ...
user avatar
4 votes
1 answer
3k views

Small number of estimators in gradient boosting

I am tuning a regression gradient boosting-based model to determine the appropriate hyperparameters using 4-folds cross validation. More specifically, I am using XGBoost and lightGBM for the models ...
user avatar
4 votes
1 answer
566 views

ROC AUC score is much less than average cross validation score

Using Lending club Dataset to find the propability of default. I am using hyperopt library to fine tune hyper parameter for an XGBclassifier and trying to maximize the ROC AUC score. I am also using ...
user avatar
3 votes
1 answer
22 views

Do i need to use hyperparamters from Gridsearch to train on WHOLE training set to get final model?

I just want to make sure i am on the right lines so please correct me if wrong. I am testing which hyperparmets are best for logisitic regession on my data X, y where X is featrues and y is target. X, ...
user avatar
  • 466