# Tag Info

9

You could fit your model/pipeline (with default parameters) to your data once and see how long it takes to train. Then you would multiply that by how many times you want to train the model through grid search. E.g. suppose you want to use a grid search to select the hyperparameters a, b and c of your pipeline. params = {'a': [1, 2, 3, 4, 5], 'b': [...

7

You could visualize them as a heatmap. For example you could use the C values as the rows, the gamma values as the columns and the color intensity of each element in the heatmap array would correspond to the mean_test_score. To implement this you first need to create a pandas.DataFrame like this:  \begin{array}{c | c c c} & C & gamma & ...

7

GridSearchCV is built around cross validation, but if speed is your main concern, you may be able to get better performance using a smaller number of folds. From the docs: class sklearn.model_selection.GridSearchCV(estimator, param_grid, scoring=None, n_jobs=None, iid='deprecated', refit=True, cv=None, verbose=0, pre_dispatch='2*n_jobs', error_score=nan, ...

6

Optimization isn't my field, but as far as I know, efficient and effective hyper-parameter optimization these days heavily revolves around building a surrogate model. As models increase in complexity, they become a more opaque black box. This is the case for deep neural nets and presumably complex trees as well. A surrogate model attempts to regress the ...

4

According to me, it is not correct to co-relate loss with accuracy. Loss is used to optimize the hypothesis such that we can get best weights whereas accuracy is used to identify how well model is doing in term of correctly predicting the values. Model internally takes the reference of predict_proba() and returns 1 if probability is > .5 otherwise 0. ...

3

As mentioned in that Kaggle notebook, you can use it pretty much as just a drop-in replacement for other search methods (grid or random). Bayesian searches still are random searches over a predefined search space/distribution, but now the algorithm pays attention to how well hyperparameter combinations perform, and will put more emphasis on high-performing ...

3

My Suggestion: The intrinsic separation of classes needs more complex model to be captured. I say this, because the difference between default model and your grid search is in max_depth parameter which is one of complexity indicators in Decision Trees. The default is None so it uses the maximum complexity it can get from max_depth but your parameter values ...

3

You can take a look at auto-sklearn. That's an automated machine learning toolkit which is a direct extension of scikit-learn.

3

When you do k-fold cross-validation, you train k models, each one of them leaving the proportion $1/k$ of the data out. For each of the models, you can compute its train error and validation error. The train error will be the error on the data selected to train the model, and the validation error will be the data left out of the training. For this reason, ...

3

If I understand, you could evaluate your approach based on your efforts in using the right preprocessing, picking the right features, building the appropriate architecture (choosing the model that best suits the problem). Each step will make the loss decrease until you reach a point, you will have to search over the hyperparameters to squeeze every percent ...

3

I have found the answer to my question : there is randomness in RandomForestRegressor(). Using the random_state hyper parameter fix the problem : rfr = RandomForestRegressor( random_state = 123 ) What is surprising is the difference of score obtained from different values of the random_state hyper parameter. It means my best hyper parameters n_estimators ...

3

Yes, you can replace the cv=5 with cv=KFold(n_splits=5, random_state=None, shuffle=False). Leaving it set to an integer, like 5, is the equivalent of setting it to either KFold(n_splits=5) or StratifiedKFold(n_splits=5), depending on the model you pass to the estimator parameter of GridSearchCV()

3

I figured out how to do that by monkey patching ParameterGrid.__iter__ and GridSearchCV._run_search methods. ParameterGrid.__iter__ iterates over all possible combinations of hyerparameters (dict of param_name: value). so i modified what it yields (one configuration of hyperparameters params) by adding "km__nbr_features" equal to 'tfidf__max_features': ...

3

By passing a callable for parameter scoring, that uses the model's oob score directly and completely ignores the passed data, you should be able to make the GridSearchCV act the way you want it to. Just pass a single split for the cv parameter, as @jncranton suggests; you can even go further and make that single split use all the data for the training ...

2

This is most likely because sklearn has chosen your majority class for you and you simply need to weigh your classes the other way around with "0" being the majority class.

2

In general, it's a good idea to split up your data into three sets: Training Set (60-80% of your data) Cross-Validation Set (10-20% of your data) Test Set (10-20% of your data) When you select a model using only a train and test set, you are selecting the model which performs the best on the test set after. This seems reasonable at first, but this is ...

2

I know this question has been here for two years, however, I was having the same problem when using cross_val_score on my data and I ended up here. The results returned from the cross_val_score function were very different from what I get when I do cross validation manually using train_test_split, as you were doing with the Nearest Neighbor classifier. ...

2

In your random forest, this is due to the fact that your final model is overfitting. Sklearn's GridSearchCV has a default argument refit = True, that takes the model with the best performance based on cross-validation and retrains it in the whole dataset. Your accuracy score is very high due to the fact that it is only measured on your training data, and the ...

2

For multi-metric evaluation, the scores for all the scorers are available in the cv_results_ dict at the keys ending with that scorer's name ('_scorer_name'). so use grid.cv_results_['mean_test_(scorer_name)'] Ex: grid.cv_results_['mean_test_r2']

2

First suggestion: you should first find a CNN architecture that satisfies you, and then stick with it. Second suggestion: be careful with cross validation. CNNs are extremely "heavy" models, they can easily take hundreds of thousands or millions of trainable parameters. Running cross validation on those could be computationally impossible. In other words, ...

2

mtry is the parameter indicating how many of the features are checked in each split decision. http://topepo.github.io/caret/train-models-by-tag.html#random-forest , see method 'rf' In Random Forest, usually more trees give more stable results, and overfitting due to number of trees is rare. Moreover, since the trees are built independently, you could just ...

2

Negative R2 values can be observed when using it in the context of model validation (where we have data that is withheld from the model) because in this context, SST $\ne$ SSE + SSR. That is, this constraint does not exist due to the data splitting. This is because in the context of model validation, the value of SST is solely calculated using the ...

2

R2 can be negative if the model is arbitrarily worse according to the sklearn documentation So the very negative train scores were indicative of an extremely bad performance. Why was the test performance so colossally bad in GridSearchCV when it was decent in a simple test/train split? The main problem is that train_test_split chooses observations randomly ...

2

You are perfectly right to pay attention to the std dev across CV folds, especially with a small dataset. As you observed, different models show different values for the performance but also for the std dev, so you have to arbitrate a tradeoff between performance and stability: The safe option is to choose the model with lower accuracy and low variance. It ...

2

I would not necessarily call it data issues. There is always some threshold that you just can not surpass, depending on the dataset ofcourse. Generally feature engineering and understanding the data will yield much greater increases than just hyp.par. optimization, which as you can see from the picture, yields often marginal increases (there is a case where ...

2

While I don't have a module named graphviz I can still try to help. Reading the documentation for GridSearchCV, I can see that there's a attribute called best_estimator_ that provides the estimator that was chosen by the search. By applying .best_estimator_ to your sample code, it seems be working fine. clf.fit(iris.data, iris.target) dot_data = ...

2

I think there are several points to take into account: First, it is possible that, in this case, the default XGBoost hyperparameters are a better combination that the ones your are passing through your params__grid combinations, you could check for it Although it does not explain your case, keep in mind that the best_score given by the GridSearchCV object ...

2

This idea of using a small sample to the data set to search for the hyperparameters is called multi-fidelity methods. A good starting point is the book by Frank Hutter, Lars Kotthoff, Joaquin Vanschoren Automated machine learning: methods, systems, challenges which is open access.

2

The GridsearchCV object in sklearn does a cross-validation on the data you feed it during your fit. In your case you have specified cv=5: this means GridSearchCV splits your data into train/test splits 5 times and reports on the mean performance over those 5 trials to be 0.868. You asked why GridSearchCV knows the best parameters without feeding it testing ...

2

I assume its crashing because not enough RAM. So I also assume your data is quite big. Your search grid is quite big. So this will definitely take some time. In order to speed up the training and overloading the RAM. You can fit the model in a subsample of data. Theoretically if your data is big enough and you sample it, when you use the whole model the ...

Only top voted, non community-wiki answers of a minimum length are eligible