11
votes
Accepted
How to estimate GridSearchCV computing time?
You could fit your model/pipeline (with default parameters) to your data once and see how long it takes to train. Then you would multiply that by how many times you want to train the model through ...
- 7,758
7
votes
What is the most efficient method for hyperparameter optimization in scikit-learn?
Optimization isn't my field, but as far as I know, efficient and effective hyper-parameter optimization these days heavily revolves around building a surrogate model. As models increase in complexity, ...
- 371
7
votes
Accepted
How to plot mean_test score and mean_train score of GridSearchCV
You could visualize them as a heatmap.
For example you could use the C values as the rows, the gamma values as the columns and ...
- 311
7
votes
GridSearch without CV
GridSearchCV is built around cross validation, but if speed is your main concern, you may be able to get better performance using a smaller number of folds.
From the docs:
class sklearn....
- 578
5
votes
Log loss vs accuracy for deciding between different learning rates?
According to me, it is not correct to co-relate loss with accuracy.
Loss is used to optimize the hypothesis such that we can get best
weights whereas accuracy is used to identify how well model ...
- 1,232
5
votes
Accepted
how to pass parameters over sklearn pipeline's stages?
I figured out how to do that by monkey patching ParameterGrid.__iter__ and GridSearchCV._run_search methods.
ParameterGrid.__iter__ iterates over all possible combinations of hyerparameters (dict of ...
- 231
5
votes
Accepted
Can GridSearchCV be used for unsupervised learning?
The goal of GridSearchCV is to iterate over (hence search) all possible combinations (hence grid) of hyper parameters and evaluate a model on a cross-validation (...
- 365
4
votes
When to use BayesianSearchCV and how it works?
As mentioned in that Kaggle notebook, you can use it pretty much as just a drop-in replacement for other search methods (grid or random). Bayesian searches still are random searches over a predefined ...
- 10.8k
4
votes
What's the default Scorer in Sci-kit learn's GridSearchCV?
From the User Guide:
By default, parameter search uses the score function of the estimator to evaluate a parameter setting. These are the ...
- 10.8k
3
votes
What is the most efficient method for hyperparameter optimization in scikit-learn?
You can take a look at auto-sklearn. That's an automated machine learning toolkit which is a direct extension of scikit-learn.
- 526
3
votes
Large negative R2 or accuracy scores for random forest with GridSearchCV but not train_test_split
Negative R2 values can be observed when using it in the context of model validation (where we have data that is withheld from the model) because in this context, SST $\ne$ SSE + SSR. That is, this ...
- 1,874
3
votes
Large negative R2 or accuracy scores for random forest with GridSearchCV but not train_test_split
R2 can be negative if the model is arbitrarily worse according to the sklearn documentation
So the very negative train scores were indicative of an extremely bad performance.
Why was the test ...
- 151
3
votes
Default parameters for decision trees give better results than parameters optimised using GridsearchCV
My Suggestion:
The intrinsic separation of classes needs more complex model to be captured. I say this, because the difference between default model and your grid search is in max_depth parameter ...
- 6,452
3
votes
Accepted
How to get mean test scores from GridSearchCV with multiple scorers - scikit-learn
For multi-metric evaluation, the scores for all the scorers are available in the cv_results_ dict at the keys ending with that scorer's name ('_scorer_name').
so use
grid.cv_results_['mean_test_(...
- 526
3
votes
Accepted
Comparison of machine learning approaches for a topic in a scientific paper
If I understand, you could evaluate your approach based on your efforts in using the right preprocessing, picking the right features, building the appropriate architecture (choosing the model that ...
- 886
3
votes
Accepted
GridSearch mean_test_score vs mean_train_score
When you do k-fold cross-validation, you train k models, each one of them leaving the proportion $1/k$ of the data out. For each of the models, you can compute its train error and validation error. ...
- 5,921
3
votes
ScikitLearn - RandomForestRegressor score different in and out of grid search
I have found the answer to my question : there is randomness in RandomForestRegressor().
Using the random_state hyper parameter fix the problem :
...
3
votes
Accepted
sklearn.model_selection: GridSearchCV vs. KFold
Yes, you can replace the cv=5 with cv=KFold(n_splits=5, random_state=None, shuffle=False). Leaving it set to an integer, like 5, ...
- 1,714
3
votes
GridSearch without CV
By passing a callable for parameter scoring, that uses the model's oob score directly and completely ignores the passed data, you should be able to make the ...
- 10.8k
3
votes
Accepted
plotting a decision tree based on gridsearchcv
While I don't have a module named graphviz I can still try to help. Reading the documentation for GridSearchCV, I can see that there's a attribute called ...
- 813
2
votes
Why is cross-validation score so low?
I know this question has been here for two years, however, I was having the same problem when using cross_val_score on my data and I ended up here.
The results ...
- 21
2
votes
Accepted
Why is cross-validation score so low?
In your random forest, this is due to the fact that your final model is overfitting. Sklearn's GridSearchCV has a default argument ...
- 5,921
2
votes
How to estimate GridSearchCV computing time?
Let the search complete and then you can use cv_results_ attribute to compute the elapsed time as given below.
...
2
votes
Class weight ineffective in sklearn
This is most likely because sklearn has chosen your majority class for you and you simply need to weigh your classes the other way around with "0" being the majority class.
- 21
2
votes
Splitting hold-out sample and training sample only once?
In general, it's a good idea to split up your data into three sets:
Training Set (60-80% of your data)
Cross-Validation Set (10-20% of your data)
Test Set (10-20% of your data)
When you select a ...
- 573
2
votes
Log loss vs accuracy for deciding between different learning rates?
Actually, loss is used by the model to decide the probability of the class. So, logloss just indicates how much is your model certain in comparison to the correct labels of the classes in test samples....
- 497
2
votes
Accepted
How to find optimal number of trees in random forest using Grid search in R?
mtry is the parameter indicating how many of the features are checked in each split decision. http://topepo.github.io/caret/train-models-by-tag.html#random-forest ,...
- 10.8k
2
votes
Accepted
Am I using GridSearch correctly or do I need to use all data for cross validation?
Your procedure is, from what I can tell, correct. You are correctly splitting your data into train/test, and then using your training data only to find optimal hyper-parameters. Using all of the ...
- 1,874
2
votes
Accepted
CNNs - Hyperparameter tuning with different training sizes of the same data set
First suggestion: you should first find a CNN architecture that satisfies you, and then stick with it.
Second suggestion: be careful with cross validation. CNNs are extremely "heavy" models, they can ...
- 5,897
2
votes
Accepted
Confusion for considering accuracy or standard deviation in selecting the best parameters
You are perfectly right to pay attention to the std dev across CV folds, especially with a small dataset. As you observed, different models show different values for the performance but also for the ...
- 24.5k
Only top scored, non community-wiki answers of a minimum length are eligible