4
$\begingroup$

When I use the hyperopt library to tune my Random Forest classifier, I get the following results:

Hyperopt estimated optimum {'max_depth': 10.0, 'n_estimators': 300.0}

However, when I train the model using its default hyperparameters, all of the evaluation metrics (Precision, Recall, F1, iba, AUC) return higher values compared to the tuned model. Should I still follow the tuned parameters? Or ignore the results of the tuning process, as it is not helping to improve the results?

Improve this question
$\endgroup$
7
  • $\begingroup$ How do you train your model? (The non hyperopt pne) Cross-validation? Do you use a validation set? It is easy to nicely overfit a random forest and get a brilliant F1 score. Whether that will actualy generalize to new data is a completely different story. $\endgroup$
    – grochmal
    May 28 '19 at 0:42
  • 2
    $\begingroup$ Thank you for the reply grochmal, I am trying both 10-fold CV and Staritifed CV. In both cases, the default values outperform the tuned parameters. Also, as my dataset is highly imbalance (and multiclass), I apply SMOTE on each fold. I do agree with the overfitting problem caused by the default values though! $\endgroup$
    – Sarah
    May 28 '19 at 1:32
  • $\begingroup$ Are the scores you are reporting the CV validation score averages? Are the folds the same for both Hyperopt and default? What hyperparameters are you searching over, and what are their distributions? Which random forest implementation are you using (esp., what are its defaults)? $\endgroup$
    – Ben Reiniger
    May 28 '19 at 1:55
  • $\begingroup$ Thank you Ben, I am using the RandomForestClassifier() from sklearn.ensemble. The folds may not be the same unless I use a 2-fold CV. I am searching for max_depth' and 'n_estimators'. The difference in the performance metrics is ~5-10%, that's why I am not sure if it's a good idea to go with the tuned model. $\endgroup$
    – Sarah
    May 28 '19 at 2:15
  • $\begingroup$ Hopefully it's not just the different folds. (Why would 2-fold produce the same folds??) sklearn's defaults are max_depth=None and n_estimators=10 [soon 100]...what ranges do you give hyperopt? (Do you only allow max_depth in 1,...,10? That would explain it...) Are the scores you're comparing the CV-scores, or an additional out-of-sample set, or ...? $\endgroup$
    – Ben Reiniger
    May 28 '19 at 2:59
1
$\begingroup$

According to me your approach of oversampling(as mentioned in comment) is not correct. When you apply SMOTE on each FOLD(assuming K-FOLD), it will over sampled the minority class. And you will always get the mixed response from K-FOLD, which is not the purpose of K-FOLD validation.

Let me try to explain you with an example:

  1. Consider you have a dataset of 4 classes A, B, C and D.
  2. Samples of each class, Class A has 10000, B has 2000, C has 1000, D has 6000.
  3. During K Fold cross validation, sample picked up by system for training having majority of A and D classes.
  4. Whereas for testing it has majorly B and C. Since you have used SMOTE it will over sampled the other two classes and you will get decent results in this case as well.

NOTE:It could be the one of the reason.

My suggestion:
 1. Shuffle the whole data set.
 2. Divide into training and test data.
 3. Apply SMOTE on training data.
 4. Follow other steps of Model training, without manipulating the data at this stage.
 5. Evaluate model on the actual test data.
Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you vipin for the intuitive example. I've tried the exact same methodology that you suggested in your answer (before going for CV). The output still follows the same behavior! Default values are outperforming the tuned parameters. I have used the Hyperopt approach to tune the RF classifier, and am running the grid search approach, will let you know if it does any better. $\endgroup$
    – Sarah
    May 28 '19 at 2:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.