Questions tagged [hyperparameter-tuning]
Hyperparameter tuning (also called hyperparameter optimization) refers to the process of finding the optimal set of hyperparameters for a given machine learning algorithm.
286
questions
0
votes
0
answers
7
views
Using "precomputed" distance matrices as input to scikit-learn clustering metrics
Is there any validity to using a distance matrix instead of the raw points with metrics such as davies_bouldin_score and ...
- 21
0
votes
1
answer
29
views
What is the prefered way to tune several hyperparameters in cross-validation? Greedy vs alltogether? [closed]
Should we tune all hyperparameters simultaneously, like in a multivariate optimization problem?
Or one after the other, in a greedy way?
The first option will produce better results but it will likely ...
- 163
0
votes
0
answers
12
views
How to get best result by using hyperparameter tuning and stacking regression?
I want to use regression ML models to get the best possible R2 score. So I decided to do both hyperparameter tuning and combining models with stacking regressor. I'm wondering if I should do ...
- 1
0
votes
0
answers
15
views
Training with optuna-tuned hyperparameters leads to different results
I'm training an image classifier in Pytorch Lightning and tuning hyperparameters with Optuna. When I use the best hyperparameters to train a separate model, the accuracies differ from those obtained ...
- 33
0
votes
0
answers
10
views
What are 'standard' parameters to try when hyperparameter tuning?
I'm curious what parameters people tend to tune for various models and the values they try. For instance, on this post, the author had a go-to set of parameters he would use for tuning.
What is the ...
- 1
0
votes
0
answers
25
views
why is there no research on machine learning algorithms to determine optimal hyperparameters for metaheuristics?
I am not shure if I am in the correct forum for this question. I'm sorry, if I'm in the wrong place here.
Question:
Why is there no research on machine learning algorithms to determine hyperparameters ...
- 101
0
votes
0
answers
23
views
Problem calculating f-1 score when when running hyperparameter tuning using Optuna
I'm trying to run Stratified CV hyperparameter tuning on an XGBClassifier using Optuna. This is a multiclass problem with 3 classes (labelled 0 through 2 in the "classes" variable). These ...
- 11
0
votes
0
answers
7
views
Test accuracy plateaus when increasing max_depth -> inf
I've built a Random Forest model that classifies into four categories based on around 10 input features.
To test the accuracy, I performed 5-fold stratified cross validation using the ...
0
votes
1
answer
27
views
Cross validation and train_test_split
I am building a class that follows the workflow:
Model Selection and Fitting
The class accepts a list of models and their respective hyperparameter grids. It then performs a standard fitting process ...
2
votes
1
answer
20
views
Changing model architecture doesn't impact results
I am currently learning binary classification.
The problem is classifying positive and negative movie reviews.
The dataset is 25,000 reviews with each review represented by 10,000 of the most used ...
- 23
0
votes
0
answers
17
views
Weights And Biases Sweep Across Multiple Datasets and Model Architectures
TLDR; How can I conduct a sweep in weights and biases across not just hyperparameters, but across model architectures and datasets as well, in a way that respects sensible aggregation and ...
- 430
0
votes
0
answers
19
views
Can somebody explain to me, how the Hyperparameters in this expamle work? I only uderstood the meaning for gamma
I am quite new to this topic, but I want to understand how Reinforcement Learning (RL) works in this example (https://gymnasium.farama.org/tutorials/training_agents/reinforce_invpend_gym_v26/). I have ...
- 1
0
votes
0
answers
6
views
How to interpet the bolded lines in bayesopt
I am using bayesopt to maximize a function that is everywhere less or equal to zero $(f(x) \leq 0)$. The score is essentially a negated Mean Absolute Error because the default behavior of bayesopt is ...
- 83
0
votes
0
answers
43
views
Build a model with cross-validation on entire dataset to learn insights?
Goal : Use XGBoost regression to learn insights from data. Prediction or forecasting not needed.
Hypothesis : If the model fits the entire dataset well, it can maybe capture its "physics" in ...
0
votes
0
answers
7
views
How to grid search LSTM manually using TimeSeriesSplit while preserve/load best_models?
This is a self-written LSTM tuner class.
...
- 1
0
votes
0
answers
20
views
Surprise NMF object is not callable
I am building a recommender system using the Sushi Preference Dataset and the NMF (Non-negative Matrix Factorization) model. I am implementing the same using the Surprise library. I want to use ...
1
vote
2
answers
39
views
which hyperparameters are returned as best in cross validation?
The description on the RandomizedSearchCV says this about best hyperparameters :
"Estimator that was chosen by the search, i.e. estimator which gave highest ...
- 31
0
votes
0
answers
21
views
How do I automatically evaluate an objective_plot after BayesSearchCV to find the *theoretical* optimal model?
I did a hyper optimization for a XGBClassifier using BayesSearchCV. I increased the kappa ...
- 101
0
votes
0
answers
47
views
Why it gives Node: 'model_8/dense_23/Relu' Matrix size-incompatible
This is my code:
...
0
votes
1
answer
51
views
Is hyperparameter tuning done on training or validation data set?
Is hyperparameter tuning done on training or validation data set? The post here gives mixed opinion as of whether the training set should be used for hyperparameter tuning. And I would like to know ...
- 419
0
votes
0
answers
146
views
Keras Tuner Error: Failed to find data adapter that can handle input: <class 'function'>, <class 'NoneType'>
I am working on a project where I have a dataset of emails with information such as subject, body, attachment names, etc. My goal is to classify/predict around ~5,000 different sites. I am using Keras ...
- 101
0
votes
0
answers
31
views
0
votes
0
answers
30
views
Walk forward cross-validation with Optuna and deepar in pytorch forecasting
I want to perform 3 splits walk forward cross validation with expanding training set for the deepar model from the pytorch forecasting framework. When I do walk forward validation, I also want to do ...
- 101
0
votes
1
answer
24
views
Flow of machine learning model including code
I'm towards the completion of my first data science project that will go into my GitHub portfolio.
I'll be happy for some clarification regarding the machine learning models section:
I got a little ...
- 15
0
votes
0
answers
29
views
Is this a valid cross validation approach to choose hyperparameters and get a good estimate of model performace?
I am using lightGBM on time series data. I first split my data set into 10% folds. The last fold is used as a test set.
For each choice of hyperparameters I first use 6 folds to train, then predict on ...
0
votes
0
answers
38
views
how does Optuna execute the objective function?
how does Optuna execute the objective function when calling study.optimize(objective, n_trials=10,
is it, for each trial ===> call the function ??, if this is the case, this means we shouldn't ...
- 101
0
votes
1
answer
53
views
Is it mandatory to set a random_state when using RandomizedSearchCV?
When I use RandomizedSearchCV, if I put the random state I always obtain the same results with the same hyperparams trainer. So, is it mandatory to use? Because in my opinion it is better to always ...
0
votes
0
answers
32
views
Is it possible to perform probability calibration with a model with the best hyperparams?
If I use RandomizedSearchCV to find the optimal hyperparams of a model, can I create another model, with those parameters, to calibrate probabilities using CalibratedClassifierCV?
The new model is not ...
0
votes
0
answers
25
views
Matlab - Machine Learning: Hyperparameter tuning of a fitcecoc - model + training it on new data
I have trained a logistic regression multi-class model in Matlab for multi-class classification using XTrainSet_A / YTrainSet_A, look at this simplified code:
...
- 1
0
votes
0
answers
29
views
how to define search space over hyperparameters automatically?
I'm trying to automate retrain steps of our ML models. My aim is hyper-parameter tuning with current data (newer performance window) using same algorithm and features on production environment. ...
- 1
1
vote
0
answers
102
views
Optuina pruning during CrossValidation, does it make sense?
I'm currently trying to build a model using CatBoost. For my parameter tuning, I'm using optuna and cross-validation and pruning the trial checking on the intermediate cross-validation scores. Here ...
1
vote
1
answer
53
views
Can I fit a model with the parameters found with RandomizedSearchCV?
I want to ask you a question.
Suppose I use the following RandomizedSearchCV to find the model's best hyperparams:
...
0
votes
0
answers
15
views
Choosing which parameters and what range to hyperparameter tune in a model?
Lets say I built an XGBoost model with XGBClassifier().
I know I can get default params by calling the default parameters function.
However, there are so many ...
- 111
0
votes
0
answers
21
views
Rules/Guidelines for Custom Weightage and Hyper-parameter tuning
I have a movie and user-ratings dataset. After implementing the content-based filtering technique, I figured, I can improvise the results even further by assigning weightage to the parameters based on ...
- 111
0
votes
0
answers
11
views
Why does cross validation and hyperparameter tuning work?
To my understanding, optimizing a model with k-fold cross validation and hyperparameter tuning are tools to be used mostly for small datasets to really make the most out of very limited/expensive data,...
- 1
0
votes
0
answers
8
views
Optimize wordembedding and neural network at the same time
I have a lot of (domain)-specific text that I want to classify into 100+ categories.
I want to train a wordembedding (FastText) and use that in conjuction with a CNN, thus I'm running into the problem ...
- 450
0
votes
1
answer
35
views
Question about grid search and KFold
I am trying an example which I am training on a huge dataset 5M (only 4 features) rows with Cudf and CUml and I am using SGD logistic regression because I must predict if the patient if is sick or not ...
- 1
1
vote
1
answer
59
views
Which of 2 options is better practice for model optimization: 1) Nested CV wrongly averaging inner CV scores. 2) Two successive CVs on X_all. Altrntv?
Goal: Compare preprocessing methods, models, and hyperparameters without leaking into the final generalization estimate, applying cross-validation (cv), i.e. NOT applying any fixed train/test splits.
...
- 11
0
votes
0
answers
58
views
Objective function in optuna seems to return at random points
I am trying to use a K-fold cross-validation within the optuna objective function.
Unfortunately, the output of the objective function seems to pop out at random places, not at the end of the cross-...
0
votes
1
answer
128
views
Tuned model has higher CV accuracy, but a lower test accuracy. Should I use the tuned or untuned model?
I am working on a classification problem using Sci Kit Learn and am confused on how to properly tune hyper parameters to get the "best" model.
Before any tuning, my logistic regression ...
0
votes
1
answer
24
views
Train/val/test approach for hyperparameter tuning
When looking to train a model, does it make sense to have a 60-20-20 train val test split, first hyper parameter tuning over the training dataset, using the validation set, picking the best model. ...
- 91
1
vote
1
answer
155
views
How to determine which combinations of parameters to include in GridSearchCV
I am using MLPClassifier from sklearn and I would like to tune it with GridSearchCV. But I don't know which set of values to include for hidden_layer_sizes, max_iter, activation, solver, etc. How can ...
- 11
0
votes
0
answers
95
views
Lightgbm model tuning produces unexpectedly different hyperparameters for similar datasets
I am trying to tune a lightgbm model for each half of a dataset that I have split by a particular feature (stock ticker in this case). Both halves have the same number of features, somewhat similar ...
0
votes
0
answers
16
views
How to deal with ambiguous classification outputs that exceed the specified threshold but are too close together?
I have a simple classification setup (intent classification). Once an input is received it's parsed using Multinomial Logistic Regression and then a score is predicted for each class. I pick the ...
- 11
0
votes
0
answers
1k
views
Got this error from Keras Tuner: Number of consecutive failures excceeded the limit of 3
I'm getting this error when I try to use Keras Tuner with my model:
Number of consecutive failures excceeded the limit of 3. .... KeyError: 'mean_squared_error'
Here's my code:
...
- 13
1
vote
1
answer
20
views
Underfitting and perfomance metrics in unsupervised methods
My question is simple and yet quite hard to find an answer to. In an unsupervised method, for example, when you have to reconstruct an input, how can you tell if your loss is good enough? Generally, ...
- 11
0
votes
1
answer
153
views
Is there any benefit to using cross validation from the XGBoost library over sklearn when tuning hyperparameters?
The XGBoost library has its own implementation of cross validation through xgboost.cv(). It looks like it requires data be stored as a DMatrix.
Instead of using <...
- 101
0
votes
2
answers
37
views
Optimization of the entire model development process
I want to perform a global optimization of the entire model development pipeline. I have several stages of development, each of which can be performed automatically: preprocessing, removal of outliers/...
- 396
1
vote
1
answer
900
views
Grid_search (RandomizedSearchCV) extremely slow with SVM (SVC)
I'm testing hyperparameters for an SVM, however, when I resort to Gridsearch or RandomizedSearchCV, I haven't been able to get a resolution, because the processing time is exceeding hours.
My dataset ...
1
vote
1
answer
276
views
Hyperparameter Tuning vs Regularization
While designing the architecture of a Neural Network, should I consider adding regularization (like Dropout, L1/L2, etc.) even after optimizing the problem using Hyperparameter Tuning? What should be ...
- 77