Questions tagged [hyperparameter]
Hyperparameters of a model are the kind of parameters that cannot be directly learned during training but are set beforehand. Hyperparameters can define, for example, the complexity of the model or its capacity to learn.
149
questions
1
vote
1
answer
26
views
Ordering of Train/Val/Test set use in hyperparameter tuning
The way I read almost lots of ML advice on these datasets sounds like "You train a model that's randomly chosen hyperparameters first on the training set, then you ignore this bit of the work, ...
- 101
0
votes
1
answer
59
views
Optimizing decision tree
I have a question regarding the technique/technology which could be applied for the issue:
Suppose I have a rule-based tree or decision tree which predicts a variable Y based on variables A,B,C. This ...
- 1
0
votes
0
answers
27
views
why is there no research on machine learning algorithms to determine optimal hyperparameters for metaheuristics?
I am not shure if I am in the correct forum for this question. I'm sorry, if I'm in the wrong place here.
Question:
Why is there no research on machine learning algorithms to determine hyperparameters ...
- 101
0
votes
0
answers
22
views
Can somebody explain to me, how the Hyperparameters in this expamle work? I only uderstood the meaning for gamma
I am quite new to this topic, but I want to understand how Reinforcement Learning (RL) works in this example (https://gymnasium.farama.org/tutorials/training_agents/reinforce_invpend_gym_v26/). I have ...
- 1
0
votes
1
answer
33
views
Visualizing correlation between hyper-parameters and metrics for Neural Network
I am working with a neural network and I want to investigate how different settings affect the loss and standard deviation of the network. I can change various parameters such as the loss function, ...
- 101
0
votes
1
answer
110
views
Is hyperparameter tuning done on training or validation data set?
Is hyperparameter tuning done on training or validation data set? The post here gives mixed opinion as of whether the training set should be used for hyperparameter tuning. And I would like to know ...
- 411
7
votes
1
answer
1k
views
Do transformers (e.g. BERT) have an unlimited input size?
There are various sources on the internet that claim that BERT has a fixed input size of 512 tokens (e.g. this, this, this, this ...). This magical number also appears in the BERT paper (Devlin et al. ...
- 193
0
votes
0
answers
18
views
Choosing which parameters and what range to hyperparameter tune in a model?
Lets say I built an XGBoost model with XGBClassifier().
I know I can get default params by calling the default parameters function.
However, there are so many ...
- 111
0
votes
0
answers
23
views
Rules/Guidelines for Custom Weightage and Hyper-parameter tuning
I have a movie and user-ratings dataset. After implementing the content-based filtering technique, I figured, I can improvise the results even further by assigning weightage to the parameters based on ...
- 111
1
vote
1
answer
82
views
Is hyperparameter tuning with different samples of data on each run a bad idea?
I have 2k time series and want to optimize the hyperparameters of my prophet model. It takes 1 hour to train and evaluate on every time series for each hyperparam combination. So, I want to run it on ...
- 268
1
vote
0
answers
78
views
Why is a neural network not doing better than multivariate linear regressions?
I am making neural networks of multiple targets, all using same training data. For some of these targets, multivariate linear regressions do a very good job, i.e. a strong linear relation exists ...
- 101
1
vote
0
answers
100
views
Relation between batch size, number of steps, and learning rate
Taking alphazero training setup as a reference:
700k total steps
batch-size of 4096
initial LR of 0.2
What would be an equivalent setup for a batch-size of 1024? ...
- 111
1
vote
1
answer
130
views
Different results between hyperparameter optimisation and actual training/val values
If I want to do a hyperparameter optimisation on a dataset using e.g. hyperband or random search, I note that some of the models being randomly chosen seem to have rather good R2 scores, MSE etc.
I ...
- 101
1
vote
0
answers
16
views
Feature Selection - Comparing Performance of different size datasets
If I have training data X, with N features, and I do feature selection, and discover n of <...
- 101
0
votes
2
answers
1k
views
Why does hyperparameter tuning occur on validation dataset and not at the very beginning?
Despite doing/using it a few times, I'm still slightly confused by the use of a validation set for hyper parameter tuning.
As far as I can tell, I choose a model, train it on training data, assess ...
- 101
0
votes
1
answer
4k
views
How to suppress "Estimator fit failed. The score on this train-test" warning message?
I am working on hyper-tuning random forest classifier with following parameters in random search CV
...
3
votes
1
answer
1k
views
Estimating Length of Hyperband Trials in Advance
I would like to use the (Keras/Tensorflow) hyperband tuning algorithm more than the Keras random search, for instance, when testing hyperparameters.
With random search I can set max trials and get a ...
- 101
0
votes
1
answer
134
views
Training Loss or Validation Loss for Hyperparameter Optimisation
When performing HO, should I be looking to train each model (each with different hyperparameter values, e.g. with RandomSearch picking those values) on the training data, and then the best one is ...
- 101
0
votes
0
answers
21
views
Scikit Learn Random Forest Classifier Hyperparameter Min Target Sample Size
From reading the docs on Scikit Learn, I haven't been able to find an answer, but does anyone know if there is a way to specify to always include a specific number out of the max sample size of ...
1
vote
1
answer
8k
views
Optimal batch size and number of epoch for BERT
I use this tutorial https://www.tensorflow.org/text/tutorials/classify_text_with_bert and get different accuracy depend on epoch numbers and batch sizes. What's optimal parameters?
- 125
1
vote
1
answer
48
views
Hyperparameter searching when there is no development set
I have a train and a test set and no development (dev) set. I'm training a model on the train set and searching for the best hyperparameters that can eventually maximize the accuracy of the test set (...
- 133
0
votes
1
answer
56
views
Can we make an AI to fine tune other AI hyper parameters?
Every time AI gurus talk about fine tuning hyper parameters, they more or less say it's trial and error. But can't we make an AI to tell AI what its hyper parameters should be?
- 101
-2
votes
1
answer
34
views
Help with Classification using scikit-learn models [closed]
I'm using the Titanic data set to classify the missing Cabins. There is a lot of missing Cabin values. My objective is just to assign the letter of the Cabin without the room number. So, I'm just ...
- 17
1
vote
1
answer
291
views
What does updated alpha mean in LDA model?
I'm trying to understand LDA model by reading through implementations of the algorithm.
Many implementations update alpha during training iterations with codes like:
...
- 11
1
vote
2
answers
518
views
select hyperparameters using Latin hypercube sampling (LHS) from a large matrix/grid of parameter combinations
I have a matrix with each row corresponds to a hyperparameter for the XGBoost model. There are seven parameters to tune in XGBoost (as shown below: nrounds/iterations, max_depth, eta, gamma, ...
- 123
1
vote
1
answer
57
views
Can I apply different hyper-parameters for different sliding time windows?
Question
Can I apply different hyper-parameters for different training sets?
I can see the point of using the shared parameters but I cannot see the point of using shared hyper-parameters. The ...
- 175
1
vote
0
answers
59
views
How to compare hyperparameter tuning in R and Python
I tried random forest in both R (Caret) and Python (Scikit-learn), but the results differ drastically.
Pearson correlation between predicted value and actual value was 0.2 in python whereas 0.8 in R.
...
- 11
1
vote
3
answers
84
views
Using Transaction Amount to Guide Learning in an Fraud Detection Machine Learning Model
I am currently using transaction amount as a feature in an XGBoost classification model designed to identify fraudulent transactions. Furthermore, transaction amount is bounded for this problem ...
- 143
1
vote
0
answers
1k
views
XGBoost: Typical gamma and min_child_weight range
What is the typical accepted range of gamma and min_child_weight parameters for the XGBoost algorithm?
Is the range of min_child_weight correlated with the number of feature or samples in the training ...
- 735
0
votes
1
answer
3k
views
BERT minimal batch size
Is there a minimum batch size for training/re-fining a BERT model on custom data?
Could you name any cases where a mini batch size between 1-8 would make sense?
Would a batch size of 1 make sense at ...
- 316
0
votes
0
answers
536
views
Hyper tuning reduce the accuracy score, why?
I have performed hyper tuning grid CV search on KNN model. The actual accuracy score for my KNN was accuracy of 42.31 % without performing hyper tuning. However, after performing hyper tuning, the ...
0
votes
1
answer
28
views
How to distinguish between different values of a hyperparameter in communication?
From Wikipedia:
In machine learning, a hyperparameter is a parameter whose value is
used to control the learning process.
If we go by the definition of parameter in What's the difference between an ...
- 125
2
votes
2
answers
320
views
Faster grid search with small dataset to derive best params instead of full dataset?
I have a dataset of 300 000 rows and an ensemble model, which include grid search to find the best params of every algorithm. Unfortunately the grid search needs to long and I have problems to ...
- 329
1
vote
0
answers
244
views
Adaptive Resampling in Caret with Pre-specified Validation Set
I was wondering if this is the correct way to get adaptive sampling in caret working with a pre-specified validation set using index. I can get this to work using the 'cv' method in caret like so
<...
- 183
0
votes
1
answer
326
views
DBSCAN Clustering
I used K-means to get the number of clusters for my data(Elbow Method). Then I was trying to see if for some specific hyperparameters can we get the same number of clusters for DBSCAN. I tried Brute-...
0
votes
1
answer
667
views
Final Model fitting - subset vs entire training data
If I used a subset of the entire available training data for model tuning and hyperparamater selection, should I fit the final model to the subset training dataset or the entire available training ...
- 735
1
vote
1
answer
2k
views
XGboost and regularization
Does the XGBClassifier method utilizes the two regularization terms reg_alpha and reg_lambda,...
- 735
1
vote
1
answer
89
views
Tuning SVM C parameter
I would like to ask for help regarding my model.
I have a dataset of preprocessed images and I performed a binary classification with SVM on Python.
I tuned the value of the c parameter from 0.001 to ...
0
votes
1
answer
673
views
Hill Climbing Algorithm - Optimum Step Size
I am implementing a standard hill climbing algorithm to optimise hyper-parameters for a predictive model. The hill climbing algorithm is being applied as part of a two-stage approach:
Apply grid ...
- 30
2
votes
3
answers
21k
views
Hyper-parameter tuning of NaiveBayes Classier
I'm fairly new to machine learning and I'm aware of the concept of hyper-parameters tuning of classifiers, and I've come across a couple of examples of this technique. However, I'm trying to use ...
- 141
9
votes
2
answers
4k
views
XGBoost and Random Forest: ntrees vs. number of boosting rounds vs. n_estimators
So I understand the main difference between Random Forests and GB Methods. Random Forests grow parallel trees and GB Methods grow one tree for each iteration. However, I am confused on the vocab used ...
- 243
1
vote
1
answer
70
views
Two questions on hyper-parameter tuning [closed]
Question 1:
In the example of logistic regression, I often see the regularization constant and penalty methods being tuned by a grid search. However, it seems like there are a lot more options for ...
- 309
0
votes
1
answer
73
views
Hyperparameter tuning of neural networks using Bayesian Optimization
One of the assumptions for finding good hyperparameters using Bayesian optimization (GP) is that the unknown function is smooth. Is this assumption valid for neural networks or at least for most of ...
- 143
1
vote
2
answers
571
views
Is the search space of Hyperparameters Continuous or Discrete?
I am looking into hyper-parameter tunning and was curious about whether the search space is considered continuous or discrete?
My understanding of both those cases:
1. Continuous would make it 'easier'...
- 11
1
vote
1
answer
38
views
Asynchronous Hyperparameter Optimization - Dependency between iterations
When using Asynchronous Hyperparameter Optimization packages such as scikit optimize or hyperopt with cross validation (e.g., cv = 2 or 4) and setting the number of iteration to N (e.g., N=100), ...
- 735
1
vote
0
answers
43
views
Classification Model showing different accuracy for SAME data?
This is my first post here, so kindly pardon any commonplace errors.
So, i have been training an XGBoost multi-class model on Google Colab. I am using a balanced dataset, with 31000 rows, where each ...
- 111
4
votes
2
answers
1k
views
Shuffle the data before splitting into folds
I am running a 4-folds cross validation hyperparameter tuning using sklearn's 'cross_validate' and 'KFold' functions. Assuming that my training dataset is already shuffled, then should I for each ...
- 735
1
vote
0
answers
30
views
Techniques for hyperparameter search in non-stationary environments
I'm tuning a supervised machine learning model over time, also called incremental learning. I do not want to assume the environment is non-stationary.
Grid search and random search do not appear to ...
- 20.5k
4
votes
1
answer
5k
views
Small number of estimators in gradient boosting
I am tuning a regression gradient boosting-based model to determine the appropriate hyperparameters using 4-folds cross validation. More specifically, I am using XGBoost and lightGBM for the models ...
- 735
4
votes
1
answer
748
views
ROC AUC score is much less than average cross validation score
Using Lending club Dataset to find the propability of default. I am using hyperopt library to fine tune hyper parameter for an XGBclassifier and trying to maximize the ROC AUC score. I am also using ...
- 53