Default parameters for decision trees give better results than parameters optimised using GridsearchCV

Question

I am using Gridsearch for a DecisionTreeClassifier predicting a binary outcome. When I run fit and predict with default parameters, I get the following results:

Accuracy: 0.9602242115860793
F1: 0.9581087077004674

Then I try GridsearchCV:

from sklearn.model_selection import GridSearchCV

param_grid = {"criterion": ["gini", "entropy"],
              "min_samples_split": [2, 10],
              "max_depth": [2, 5, 10]
              }

grid = GridSearchCV(dtc, param_grid, cv=3, scoring='neg_mean_squared_error')


grid.fit(X_train, y_train.values.ravel())

y_pred_class = grid.predict(X_test)

When I check the results in y_pred, they only contain one class (0) and thus I get a warning when I try to look at F1:

site-packages\sklearn\metrics\classification.py:1135: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples. 'precision', 'predicted', average, warn_for)

Could anyone please suggest what could be the issue here and why the best parameters give the same prediction for the whole set?

Answer: With that many features in my dataset, restricting max depth and min samples split with the specified values could not give adequate results. Adding None and increasing the range fixed the issue.

Answer 1

My Suggestion:

The intrinsic separation of classes needs more complex model to be captured. I say this, because the difference between default model and your grid search is in max_depth parameter which is one of complexity indicators in Decision Trees. The default is None so it uses the maximum complexity it can get from max_depth but your parameter values are at most 10. To check this, you may try increasing the max_depth in your grid search (or leave it None) and see the result of grid search. If it improves then this is the point.

Answer 2

The result you get from grid search is the best parameter from that range given in grid search and then all such cases are tested against the mentioned performance parameters(accuracy,AUC,precision, F1 Score,etc). The best one is selected.

The result you got is the best from the mentioned parameters range. You might get better results by changing the range of the values for each hyper parameters.

One of the approach I used is two way approach. Firstly, takes the longer range for the parameters and do the coarse search where you get and idea about approx range that needs to be given for the parameters search.

Secondly, do the grid search but this time with fine search by selecting the range from the intuitions and approximation got from first step.

Likewise, you will get optimum values for all parameters that helps to tune your model.

Answer 3

Try first RandomizedSearchCV to narrow the range of search. Then you can use GridsearchCV trying parameters around the found range for the best estimator.

Answer 4

I saw a similar Problem on StackOverflow.

According to the sklearn Documentation, UndefinedMetricWarning is raised when TP + FP == 0 or TP + FN == 0. So in your CV fold you encounter either the case where your model does not predict positives or that there are no positives in the fold.

For your question about the performance, I'd encourage you to broaden the parameters that are searched in your grid search and including the default parameters, so you are always better than not using grid search at all.

Additionally, I'd suggest you take a look into randomized grid search, where not all parameters are tried out, but rather a fixed number of parameter settings. By this you can limit the time spent for your grid search, but do not run into the problem that you might "miss out" a very good parameter setting. Take a look at this blog post for more detailed information on GS VS Randomized GS.