Questions tagged [hyperparameter-tuning]

Hyperparameter tuning (also called hyperparameter optimization) refers to the process of finding the optimal set of hyperparameters for a given machine learning algorithm.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

Using "precomputed" distance matrices as input to scikit-learn clustering metrics

Is there any validity to using a distance matrix instead of the raw points with metrics such as davies_bouldin_score and ...
Chris Coffee's user avatar
0 votes
1 answer
29 views

What is the prefered way to tune several hyperparameters in cross-validation? Greedy vs alltogether? [closed]

Should we tune all hyperparameters simultaneously, like in a multivariate optimization problem? Or one after the other, in a greedy way? The first option will produce better results but it will likely ...
skan's user avatar
  • 163
0 votes
0 answers
12 views

How to get best result by using hyperparameter tuning and stacking regression?

I want to use regression ML models to get the best possible R2 score. So I decided to do both hyperparameter tuning and combining models with stacking regressor. I'm wondering if I should do ...
User's user avatar
  • 1
0 votes
0 answers
15 views

Training with optuna-tuned hyperparameters leads to different results

I'm training an image classifier in Pytorch Lightning and tuning hyperparameters with Optuna. When I use the best hyperparameters to train a separate model, the accuracies differ from those obtained ...
Tirtha's user avatar
  • 33
0 votes
0 answers
10 views

What are 'standard' parameters to try when hyperparameter tuning?

I'm curious what parameters people tend to tune for various models and the values they try. For instance, on this post, the author had a go-to set of parameters he would use for tuning. What is the ...
Marsha T's user avatar
0 votes
0 answers
25 views

why is there no research on machine learning algorithms to determine optimal hyperparameters for metaheuristics?

I am not shure if I am in the correct forum for this question. I'm sorry, if I'm in the wrong place here. Question: Why is there no research on machine learning algorithms to determine hyperparameters ...
Andre's user avatar
  • 101
0 votes
0 answers
23 views

Problem calculating f-1 score when when running hyperparameter tuning using Optuna

I'm trying to run Stratified CV hyperparameter tuning on an XGBClassifier using Optuna. This is a multiclass problem with 3 classes (labelled 0 through 2 in the "classes" variable). These ...
Metrician's user avatar
0 votes
0 answers
7 views

Test accuracy plateaus when increasing max_depth -> inf

I've built a Random Forest model that classifies into four categories based on around 10 input features. To test the accuracy, I performed 5-fold stratified cross validation using the ...
okjdlsksjdwi's user avatar
0 votes
1 answer
27 views

Cross validation and train_test_split

I am building a class that follows the workflow: Model Selection and Fitting The class accepts a list of models and their respective hyperparameter grids. It then performs a standard fitting process ...
Guilherme Raibolt's user avatar
2 votes
1 answer
20 views

Changing model architecture doesn't impact results

I am currently learning binary classification. The problem is classifying positive and negative movie reviews. The dataset is 25,000 reviews with each review represented by 10,000 of the most used ...
Omer Mualem's user avatar
0 votes
0 answers
17 views

Weights And Biases Sweep Across Multiple Datasets and Model Architectures

TLDR; How can I conduct a sweep in weights and biases across not just hyperparameters, but across model architectures and datasets as well, in a way that respects sensible aggregation and ...
Warlax56's user avatar
  • 430
0 votes
0 answers
19 views

Can somebody explain to me, how the Hyperparameters in this expamle work? I only uderstood the meaning for gamma

I am quite new to this topic, but I want to understand how Reinforcement Learning (RL) works in this example (https://gymnasium.farama.org/tutorials/training_agents/reinforce_invpend_gym_v26/). I have ...
Peter's user avatar
  • 1
0 votes
0 answers
6 views

How to interpet the bolded lines in bayesopt

I am using bayesopt to maximize a function that is everywhere less or equal to zero $(f(x) \leq 0)$. The score is essentially a negated Mean Absolute Error because the default behavior of bayesopt is ...
Enk9456's user avatar
  • 83
0 votes
0 answers
43 views

Build a model with cross-validation on entire dataset to learn insights?

Goal : Use XGBoost regression to learn insights from data. Prediction or forecasting not needed. Hypothesis : If the model fits the entire dataset well, it can maybe capture its "physics" in ...
cwanderroycbooks's user avatar
0 votes
0 answers
7 views

How to grid search LSTM manually using TimeSeriesSplit while preserve/load best_models?

This is a self-written LSTM tuner class. ...
frr0717's user avatar
0 votes
0 answers
20 views

Surprise NMF object is not callable

I am building a recommender system using the Sushi Preference Dataset and the NMF (Non-negative Matrix Factorization) model. I am implementing the same using the Surprise library. I want to use ...
Sumant Chopde's user avatar
1 vote
2 answers
39 views

which hyperparameters are returned as best in cross validation?

The description on the RandomizedSearchCV says this about best hyperparameters : "Estimator that was chosen by the search, i.e. estimator which gave highest ...
pppp_prs's user avatar
0 votes
0 answers
21 views

How do I automatically evaluate an objective_plot after BayesSearchCV to find the *theoretical* optimal model?

I did a hyper optimization for a XGBClassifier using BayesSearchCV. I increased the kappa ...
Jack Sabbath's user avatar
0 votes
0 answers
47 views

Why it gives Node: 'model_8/dense_23/Relu' Matrix size-incompatible

This is my code: ...
Tay Kim Gaik's user avatar
0 votes
1 answer
51 views

Is hyperparameter tuning done on training or validation data set?

Is hyperparameter tuning done on training or validation data set? The post here gives mixed opinion as of whether the training set should be used for hyperparameter tuning. And I would like to know ...
Student's user avatar
  • 419
0 votes
0 answers
146 views

Keras Tuner Error: Failed to find data adapter that can handle input: <class 'function'>, <class 'NoneType'>

I am working on a project where I have a dataset of emails with information such as subject, body, attachment names, etc. My goal is to classify/predict around ~5,000 different sites. I am using Keras ...
patchie's user avatar
  • 101
0 votes
0 answers
31 views

Hyperparameters values changing on every run

...
Akshita's user avatar
0 votes
0 answers
30 views

Walk forward cross-validation with Optuna and deepar in pytorch forecasting

I want to perform 3 splits walk forward cross validation with expanding training set for the deepar model from the pytorch forecasting framework. When I do walk forward validation, I also want to do ...
Jose_Peeterson's user avatar
0 votes
1 answer
24 views

Flow of machine learning model including code

I'm towards the completion of my first data science project that will go into my GitHub portfolio. I'll be happy for some clarification regarding the machine learning models section: I got a little ...
Sigal Cohen's user avatar
0 votes
0 answers
29 views

Is this a valid cross validation approach to choose hyperparameters and get a good estimate of model performace?

I am using lightGBM on time series data. I first split my data set into 10% folds. The last fold is used as a test set. For each choice of hyperparameters I first use 6 folds to train, then predict on ...
user3376601's user avatar
0 votes
0 answers
38 views

how does Optuna execute the objective function?

how does Optuna execute the objective function when calling study.optimize(objective, n_trials=10, is it, for each trial ===> call the function ??, if this is the case, this means we shouldn't ...
Kaoutar's user avatar
  • 101
0 votes
1 answer
53 views

Is it mandatory to set a random_state when using RandomizedSearchCV?

When I use RandomizedSearchCV, if I put the random state I always obtain the same results with the same hyperparams trainer. So, is it mandatory to use? Because in my opinion it is better to always ...
Flavio Brienza's user avatar
0 votes
0 answers
32 views

Is it possible to perform probability calibration with a model with the best hyperparams?

If I use RandomizedSearchCV to find the optimal hyperparams of a model, can I create another model, with those parameters, to calibrate probabilities using CalibratedClassifierCV? The new model is not ...
Flavio Brienza's user avatar
0 votes
0 answers
25 views

Matlab - Machine Learning: Hyperparameter tuning of a fitcecoc - model + training it on new data

I have trained a logistic regression multi-class model in Matlab for multi-class classification using XTrainSet_A / YTrainSet_A, look at this simplified code: ...
De Ro's user avatar
  • 1
0 votes
0 answers
29 views

how to define search space over hyperparameters automatically?

I'm trying to automate retrain steps of our ML models. My aim is hyper-parameter tuning with current data (newer performance window) using same algorithm and features on production environment. ...
Atacan's user avatar
  • 1
1 vote
0 answers
102 views

Optuina pruning during CrossValidation, does it make sense?

I'm currently trying to build a model using CatBoost. For my parameter tuning, I'm using optuna and cross-validation and pruning the trial checking on the intermediate cross-validation scores. Here ...
GiusWestsideDS's user avatar
1 vote
1 answer
53 views

Can I fit a model with the parameters found with RandomizedSearchCV?

I want to ask you a question. Suppose I use the following RandomizedSearchCV to find the model's best hyperparams: ...
Flavio Brienza's user avatar
0 votes
0 answers
15 views

Choosing which parameters and what range to hyperparameter tune in a model?

Lets say I built an XGBoost model with XGBClassifier(). I know I can get default params by calling the default parameters function. However, there are so many ...
Katsu's user avatar
  • 111
0 votes
0 answers
21 views

Rules/Guidelines for Custom Weightage and Hyper-parameter tuning

I have a movie and user-ratings dataset. After implementing the content-based filtering technique, I figured, I can improvise the results even further by assigning weightage to the parameters based on ...
shripal mehta's user avatar
0 votes
0 answers
11 views

Why does cross validation and hyperparameter tuning work?

To my understanding, optimizing a model with k-fold cross validation and hyperparameter tuning are tools to be used mostly for small datasets to really make the most out of very limited/expensive data,...
frankL's user avatar
  • 1
0 votes
0 answers
8 views

Optimize wordembedding and neural network at the same time

I have a lot of (domain)-specific text that I want to classify into 100+ categories. I want to train a wordembedding (FastText) and use that in conjuction with a CNN, thus I'm running into the problem ...
CutePoison's user avatar
0 votes
1 answer
35 views

Question about grid search and KFold

I am trying an example which I am training on a huge dataset 5M (only 4 features) rows with Cudf and CUml and I am using SGD logistic regression because I must predict if the patient if is sick or not ...
gkasap's user avatar
  • 1
1 vote
1 answer
59 views

Which of 2 options is better practice for model optimization: 1) Nested CV wrongly averaging inner CV scores. 2) Two successive CVs on X_all. Altrntv?

Goal: Compare preprocessing methods, models, and hyperparameters without leaking into the final generalization estimate, applying cross-validation (cv), i.e. NOT applying any fixed train/test splits. ...
le8rning's user avatar
0 votes
0 answers
58 views

Objective function in optuna seems to return at random points

I am trying to use a K-fold cross-validation within the optuna objective function. Unfortunately, the output of the objective function seems to pop out at random places, not at the end of the cross-...
Adrian Mureșan's user avatar
0 votes
1 answer
128 views

Tuned model has higher CV accuracy, but a lower test accuracy. Should I use the tuned or untuned model?

I am working on a classification problem using Sci Kit Learn and am confused on how to properly tune hyper parameters to get the "best" model. Before any tuning, my logistic regression ...
d0dg3r_k1d's user avatar
0 votes
1 answer
24 views

Train/val/test approach for hyperparameter tuning

When looking to train a model, does it make sense to have a 60-20-20 train val test split, first hyper parameter tuning over the training dataset, using the validation set, picking the best model. ...
Socorro's user avatar
  • 91
1 vote
1 answer
155 views

How to determine which combinations of parameters to include in GridSearchCV

I am using MLPClassifier from sklearn and I would like to tune it with GridSearchCV. But I don't know which set of values to include for hidden_layer_sizes, max_iter, activation, solver, etc. How can ...
Penjan's user avatar
  • 11
0 votes
0 answers
95 views

Lightgbm model tuning produces unexpectedly different hyperparameters for similar datasets

I am trying to tune a lightgbm model for each half of a dataset that I have split by a particular feature (stock ticker in this case). Both halves have the same number of features, somewhat similar ...
Generalenthu's user avatar
0 votes
0 answers
16 views

How to deal with ambiguous classification outputs that exceed the specified threshold but are too close together?

I have a simple classification setup (intent classification). Once an input is received it's parsed using Multinomial Logistic Regression and then a score is predicted for each class. I pick the ...
Metrician's user avatar
0 votes
0 answers
1k views

Got this error from Keras Tuner: Number of consecutive failures excceeded the limit of 3

I'm getting this error when I try to use Keras Tuner with my model: Number of consecutive failures excceeded the limit of 3. .... KeyError: 'mean_squared_error' Here's my code: ...
soilwatch's user avatar
1 vote
1 answer
20 views

Underfitting and perfomance metrics in unsupervised methods

My question is simple and yet quite hard to find an answer to. In an unsupervised method, for example, when you have to reconstruct an input, how can you tell if your loss is good enough? Generally, ...
BilboBuggins's user avatar
0 votes
1 answer
153 views

Is there any benefit to using cross validation from the XGBoost library over sklearn when tuning hyperparameters?

The XGBoost library has its own implementation of cross validation through xgboost.cv(). It looks like it requires data be stored as a DMatrix. Instead of using <...
Eli's user avatar
  • 101
0 votes
2 answers
37 views

Optimization of the entire model development process

I want to perform a global optimization of the entire model development pipeline. I have several stages of development, each of which can be performed automatically: preprocessing, removal of outliers/...
Andrew's user avatar
  • 396
1 vote
1 answer
900 views

Grid_search (RandomizedSearchCV) extremely slow with SVM (SVC)

I'm testing hyperparameters for an SVM, however, when I resort to Gridsearch or RandomizedSearchCV, I haven't been able to get a resolution, because the processing time is exceeding hours. My dataset ...
Paulo Sergio Moreira's user avatar
1 vote
1 answer
276 views

Hyperparameter Tuning vs Regularization

While designing the architecture of a Neural Network, should I consider adding regularization (like Dropout, L1/L2, etc.) even after optimizing the problem using Hyperparameter Tuning? What should be ...
Harsh Khare's user avatar

1
2 3 4 5 6