10 votes
Accepted

How to estimate GridSearchCV computing time?

You could fit your model/pipeline (with default parameters) to your data once and see how long it takes to train. Then you would multiply that by how many times you want to train the model through ...
  • 7,698
7 votes

What is the most efficient method for hyperparameter optimization in scikit-learn?

Optimization isn't my field, but as far as I know, efficient and effective hyper-parameter optimization these days heavily revolves around building a surrogate model. As models increase in complexity, ...
  • 371
7 votes
Accepted

How to plot mean_test score and mean_train score of GridSearchCV

You could visualize them as a heatmap. For example you could use the C values as the rows, the gamma values as the columns and ...
  • 311
7 votes

GridSearch without CV

GridSearchCV is built around cross validation, but if speed is your main concern, you may be able to get better performance using a smaller number of folds. From the docs: class sklearn....
  • 578
5 votes

Log loss vs accuracy for deciding between different learning rates?

According to me, it is not correct to co-relate loss with accuracy. Loss is used to optimize the hypothesis such that we can get best weights whereas accuracy is used to identify how well model ...
  • 1,234
5 votes
Accepted

how to pass parameters over sklearn pipeline's stages?

I figured out how to do that by monkey patching ParameterGrid.__iter__ and GridSearchCV._run_search methods. ParameterGrid.__iter__ iterates over all possible combinations of hyerparameters (dict of ...
5 votes
Accepted

Can GridSearchCV be used for unsupervised learning?

The goal of GridSearchCV is to iterate over (hence search) all possible combinations (hence grid) of hyper parameters and evaluate a model on a cross-validation (...
4 votes

When to use BayesianSearchCV and how it works?

As mentioned in that Kaggle notebook, you can use it pretty much as just a drop-in replacement for other search methods (grid or random). Bayesian searches still are random searches over a predefined ...
  • 10.6k
4 votes

What's the default Scorer in Sci-kit learn's GridSearchCV?

From the User Guide: By default, parameter search uses the score function of the estimator to evaluate a parameter setting. These are the ...
  • 10.6k
3 votes

What is the most efficient method for hyperparameter optimization in scikit-learn?

You can take a look at auto-sklearn. That's an automated machine learning toolkit which is a direct extension of scikit-learn.
  • 526
3 votes

Large negative R2 or accuracy scores for random forest with GridSearchCV but not train_test_split

Negative R2 values can be observed when using it in the context of model validation (where we have data that is withheld from the model) because in this context, SST $\ne$ SSE + SSR. That is, this ...
  • 1,864
3 votes

Large negative R2 or accuracy scores for random forest with GridSearchCV but not train_test_split

R2 can be negative if the model is arbitrarily worse according to the sklearn documentation So the very negative train scores were indicative of an extremely bad performance. Why was the test ...
  • 151
3 votes

Default parameters for decision trees give better results than parameters optimised using GridsearchCV

My Suggestion: The intrinsic separation of classes needs more complex model to be captured. I say this, because the difference between default model and your grid search is in max_depth parameter ...
3 votes
Accepted

How to get mean test scores from GridSearchCV with multiple scorers - scikit-learn

For multi-metric evaluation, the scores for all the scorers are available in the cv_results_ dict at the keys ending with that scorer's name ('_scorer_name'). so use grid.cv_results_['mean_test_(...
  • 526
3 votes
Accepted

Comparison of machine learning approaches for a topic in a scientific paper

If I understand, you could evaluate your approach based on your efforts in using the right preprocessing, picking the right features, building the appropriate architecture (choosing the model that ...
3 votes
Accepted

GridSearch mean_test_score vs mean_train_score

When you do k-fold cross-validation, you train k models, each one of them leaving the proportion $1/k$ of the data out. For each of the models, you can compute its train error and validation error. ...
  • 5,891
3 votes

ScikitLearn - RandomForestRegressor score different in and out of grid search

I have found the answer to my question : there is randomness in RandomForestRegressor(). Using the random_state hyper parameter fix the problem : ...
3 votes
Accepted

sklearn.model_selection: GridSearchCV vs. KFold

Yes, you can replace the cv=5 with cv=KFold(n_splits=5, random_state=None, shuffle=False). Leaving it set to an integer, like 5, ...
  • 1,714
3 votes

GridSearch without CV

By passing a callable for parameter scoring, that uses the model's oob score directly and completely ignores the passed data, you should be able to make the ...
  • 10.6k
2 votes

Why is cross-validation score so low?

I know this question has been here for two years, however, I was having the same problem when using cross_val_score on my data and I ended up here. The results ...
  • 21
2 votes
Accepted

Why is cross-validation score so low?

In your random forest, this is due to the fact that your final model is overfitting. Sklearn's GridSearchCV has a default argument ...
  • 5,891
2 votes

How to estimate GridSearchCV computing time?

Let the search complete and then you can use cv_results_ attribute to compute the elapsed time as given below. ...
2 votes

Class weight ineffective in sklearn

This is most likely because sklearn has chosen your majority class for you and you simply need to weigh your classes the other way around with "0" being the majority class.
  • 21
2 votes

Splitting hold-out sample and training sample only once?

In general, it's a good idea to split up your data into three sets: Training Set (60-80% of your data) Cross-Validation Set (10-20% of your data) Test Set (10-20% of your data) When you select a ...
2 votes

Log loss vs accuracy for deciding between different learning rates?

Actually, loss is used by the model to decide the probability of the class. So, logloss just indicates how much is your model certain in comparison to the correct labels of the classes in test samples....
2 votes
Accepted

How to find optimal number of trees in random forest using Grid search in R?

mtry is the parameter indicating how many of the features are checked in each split decision. http://topepo.github.io/caret/train-models-by-tag.html#random-forest ,...
  • 10.6k
2 votes
Accepted

Am I using GridSearch correctly or do I need to use all data for cross validation?

Your procedure is, from what I can tell, correct. You are correctly splitting your data into train/test, and then using your training data only to find optimal hyper-parameters. Using all of the ...
  • 1,864
2 votes
Accepted

CNNs - Hyperparameter tuning with different training sizes of the same data set

First suggestion: you should first find a CNN architecture that satisfies you, and then stick with it. Second suggestion: be careful with cross validation. CNNs are extremely "heavy" models, they can ...
  • 5,797
2 votes
Accepted

Confusion for considering accuracy or standard deviation in selecting the best parameters

You are perfectly right to pay attention to the std dev across CV folds, especially with a small dataset. As you observed, different models show different values for the performance but also for the ...
  • 24.4k
2 votes
Accepted

How to yield better AUC score?

I would not necessarily call it data issues. There is always some threshold that you just can not surpass, depending on the dataset ofcourse. Generally feature engineering and understanding the data ...
  • 5,459

Only top scored, non community-wiki answers of a minimum length are eligible